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A  COST-BENEFIT  ANALYSIS  APPLIED  TO  EXAMPLE  PROPOSALS  FOR  ARMY 
TRAINING  AND  EDUCATION  RESEARCH 


EXECUTIVE  SUMMARY  

Background  and  Requirement: 

This  document  presents  a  methodology  for  assessing  the  value  of  research  and  development 
(R&D)  opportunities  for  Army  training  and  education.  The  methodology  uses  cost-benefit 
analysis  to  determine  potential  payoffs  of  learning  science  R&D.  It  is  applied  to  ideas  derived 
from  the  Army  Science  of  Learning  Workshop  sponsored  by  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  at  the  request  of  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC).  The  workshop  brought  together  key  stakeholders  in  Army 
training  and  education,  experts  from  academia  and  industry,  and  representatives  of  other 
Services.  The  workshop’s  participants  identified  learning  science  findings  and  technologies  to 
help  the  Army  train  Soldiers  and  grow  leaders  for  today  and  tomorrow. 

The  workshop’s  participants  were  assigned  to  one  of  four  working  groups  (Learning  Model, 
Train  Soldiers,  Develop  Leaders,  and  Future  Capabilities)  based  on  their  expertise.  Each  working 
group  was  chaired  by  a  facilitator,  who  was  a  senior  ARI  researcher,  and  assisted  by  a 
stakeholder,  who  represented  TRADOC  interests.  The  working  group  chair  nominated 
participants  for  their  respective  groups  and  developed  the  objectives  for  their  activities,  which 
are  described  as  follows: 

•  Learning  Model.  This  group  examined  learning  models  of  other  Services  and 
organizations  to  extract  best  practices  and  lessons  learned  that  could  be  applicable  for 
an  Army  learning  model. 

•  Train  Soldiers.  This  group  examined  the  reshaping  of  the  Army’s  training  and 
education  system.  It  focused  on  how  advances  in  the  science  of  learning  have 
provided  new  pedagogical  techniques,  procedures,  and  technologies  that  can  be  used 
to  offset  pending  schoolhouse  resource  reductions. 

•  Develop  Leaders.  This  group  examined  what  we  know  about  “adaptability”  and 
explored  possible  strategies  for  accelerating  the  growth  of  key  leader  skills  that  are 
thought  to  support  adaptability.  It  also  discussed  some  of  the  organizational  factors 
(socialization,  cohesion)  that  help  establish  the  climate  within  which  adaptable 
leaders  perform. 

•  Future  Capabilities.  This  group’s  goal  was  to  envision  what  learning  science  and 
emerging  technologies  will  offer  in  the  next  decade  and  to  articulate  how  the  Army 
might  take  advantage  of  these  technologies  to  train  Soldiers  and  grow  leaders  more 
effectively. 


v 


Procedure: 


The  analysis  methodology  provides  estimates  of  the  costs  and  benefits  of  R&D  projects  for 
TRADOC  proponency.  The  intent  is  to  produce  a  generic  approach  for  assessing  and  planning 
research — not  just  for  application  to  the  current  set  of  proposals,  but  also  for  use  with  proposals 
modified  from  the  present  set  or  with  completely  new  ones.  The  approach  is  organized  around 
three  analysis  components:  proposals,  cost  estimates,  and  benefit  estimates. 

1.  Proposals.  The  elements  of  the  current  analysis  are  21  proposed  R&D  efforts  compiled 
by  the  Institute  for  Defense  Analyses  (IDA)  based  on  some  of  the  more  important 
concepts  that  evolved  from  the  workshop’s  sessions.  Each  proposal  contains  basic 
concepts  and  estimates  of  the  required  time  and  personnel  resources. 

2.  Cost  Estimates.  The  proposal  writers  provided  cost  estimates,  which  were  decomposed 
into  implementation  costs,  maintenance  costs,  and  other  costs  (e.g.,  computer  hardware 
and  software).  Total  costs  were  calculated  in  two  ways.  First,  implementation  and  other 
costs  were  summed  to  estimate  the  first-year  (Yl)  start-up  costs.  Second,  a  more  long¬ 
term  estimate  was  calculated  by  adding  maintenance  to  Yl  costs,  assuming  a  5-year  time 
frame  to  correspond  with  standard  Future  Years  Defense  Plan  (FYDP)  estimates. 

3.  Benefit  Estimates.  The  benefit  of  a  proposed  R&D  effort  was  conceived  as  being 
analogous  to  expected  value  (i.e.,  an  estimate  of  the  work’s  operational  impact  multiplied 
by  the  probability  of  success).  Proposal  writers  estimated  the  impact  of  each  proposed 
effort,  with  the  resulting  transformed  consensus  rating  ranging  from  1 0  (least  important) 
to  100  (most  important).  They  also  estimated  the  probability  of  success  principally  as  a 
function  of  the  proposal’s  technical  feasibility,  although  some  consideration  was  given  to 
the  practical  constraints  of  executing  the  research. 

Findings: 

Benefit-to-cost  ratios  based  on  both  Yl  and  FYDP  costs  were  calculated  for  each  of  21  R&D 
opportunuities  derived  from  the  workshop.  The  proposals  were  sorted  by  the  ratios,  and 
incremental  benefit  was  plotted  as  a  function  of  incremental  Yl  and  incremental  FYDP  costs. 

The  results  showed  that  the  method  is  sensitive  to  key  aspects  of  cost-benefit  analysis  including 
the  following  three  major  categories: 

1 .  Maximize  benefits/minimize  cost.  An  optimal  point  that  balances  benefit  and  cost. 

2.  Fixed  cost.  For  a  fixed  budget,  the  function  defines  a  set  of  R&D  proposals  that 
maximizes  total  benefit. 

3.  Fixed  benefit.  For  a  minimum  standard  of  benefit  (e.g.,  80%),  the  function  defines  a  set  of 
R&D  proposals  that  minimize  costs. 

To  illustrate  the  process,  consider  a  package  that  contains  a  single  proposal.  The  optimal 
selection  would  be  the  proposal  with  the  greatest  benefit-to-cost  ratio.  The  optimal  two-proposal 
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package  would  then  include  that  same  proposal  plus  the  one  with  the  next  highest  benefit-to-cost 
ratio.  By  ordering  the  proposals  in  this  way,  one  can  calculate  the  total  benefit  of  the  proposal 
packages  as  a  function  of  their  cumulative  costs.  More  generally,  this  function  reveals  packages 
of  proposals  that  optimize  specific  benefit  and  cost  levels  and  that  are  sensitive  to  using  Y1  vs. 
FYDP  cost  definitions.  Details  with  extensive  examples  are  presented  in  the  document. 

Utilization  and  Dissemination  of  Findings: 

The  principal  finding  of  this  work  is  that  application  of  cost-benefit  analyses  produced  sensible 
alternative  plans  for  a  TRADOC  R&D  program.  The  costing  components  of  the  methodology  are 
well-understood  traditional  elements  of  cost-benefit  analyses.  The  innovative  aspect  of  the 
method — the  benefits  components — had  a  notable  role  in  its  apparent  success.  The  innovation 
was  to  quantify  benefit  by  parsing  it  into  two  subcomponents:  impact  of  the  effort  and  the 
probability  of  its  success.  The  multiplicative  combination  of  these  subcomponents  was 
interpreted  as  the  “expected  impact”  of  the  proposed  R&D  efforts. 

Given  credible  inputs,  this  document  demonstrates  how  to  integrate  R&D  proposals  with  an 
analysis  methodology  to  determine  their  value  to  the  Army.  With  that  in  mind,  TRADOC  may 
wish  to  alter  the  cost  or  benefit  values  and  rerun  the  analysis  on  the  proposed  R&D  projects.  In 
addition,  users  of  the  methodology  may  want  to  modify  the  objects  of  analysis  (i.e.,  the  R&D 
proposals).  Please  note  that  performing  the  necessary  judgments  and  achieving  a  consensus  of 
opinion  are  themselves  nontrivial  exercises. 
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A  COST-BENEFIT  ANALYSIS  APPLIED  TO  EXAMPLE  PROPOSALS 
FOR  ARMY  TRAINING  AND  EDUCATION  RESEARCH 

Introduction 


Background 

The  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  is  redefining  its  role  as 
“Architect  of  the  Army”  to  better  support  and  shape  the  Operational  Force.  In  that  role, 
TRADOC  is  looking  outside  traditional  organizational  relationships  to  gain  a  better 
understanding  of  the  fundamental  processes  that  underlie  learning  and  to  identify  state-of-the-art 
technologies  that  enhance  learning. 

To  help  TRADOC  understand  these  issues,  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI)  sponsored  the  Army  Science  of  Learning  Workshop  on 
1-3  August  2006  in  Hampton,  Virginia  (Quinkert,  Morrison,  Fletcher,  Moses,  and  Roberts, 
2007).  The  workshop  brought  together  key  stakeholders  in  Army  training  and  education,  experts 
from  academia  and  industry,  and  representatives  of  other  Services.  The  overall  purpose  of  this 
workshop  was  to  identify  learning  science  findings  and  technologies  to  help  the  Army  train 
Soldiers  and  develop  leaders  for  today  and  tomorrow. 

The  workshop’s  participants  were  assigned  to  one  of  four  working  groups  (Learning 
Model,  Train  Soldiers,  Develop  Leaders,  and  Future  Capabilities)  based  on  their  expertise.  Each 
working  group  was  chaired  by  a  facilitator,  who  was  a  senior  ARI  researcher,  and  assisted  by  a 
stakeholder,  who  represented  TRADOC  interests.  The  working  group  chair  nominated 
participants  for  their  respective  groups  and  developed  the  objectives  for  their  activities,  which 
are  described  as  follows: 

•  Learning  Model.  This  group  examined  learning  models  of  other  Services  and 
organizations  to  extract  best  practices  and  lessons  learned  that  could  be  applicable  for 
an  Army  learning  model. 

•  Train  Soldiers.  This  group  examined  the  reshaping  of  the  Army’s  training  and 
education  system.  It  focused  on  how  advances  in  the  science  of  learning  have 
provided  new  pedagogical  techniques,  procedures,  and  technologies  that  can  be  used 
to  offset  pending  schoolhouse  resource  reductions. 

•  Develop  Leaders.  This  group  examined  what  we  know  about  “adaptability”  and 
explored  possible  strategies  for  accelerating  the  growth  of  key  leader  skills  that  are 
thought  to  support  adaptability.  It  also  discussed  some  of  the  organizational  factors 
(socialization,  cohesion)  that  help  to  establish  the  climate  within  which  adaptable 
leaders  perform. 
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•  Future  Capabilities.  This  group’s  goal  was  to  envision  what  learning  science  and 
emerging  technologies  will  offer  within  the  next  decade  and  articulate  how  the  Army 
might  take  advantage  of  these  technologies  to  train  Soldiers  and  grow  leaders  more 
effectively. 

Organization  and  Objective  of  This  Report 

This  report  presents  specific  proposals  for  research  and  development  (R&D)  that  were 
derived  from  the  Army  workshop.  The  current  analysis  was  done  to  estimate  the  costs  and 
benefits  of  R&D  projects  for  TRADOC  proponency.  This  report  also  documents  the 
development  approach  for  making  programmatic  decisions  about  R&D  opportunities.  The  intent 
is  to  provide  a  generic  approach  for  assessing  and  planning  research — not  just  for  application  to 
the  current  set  of  proposals,  but  also  for  use  with  proposals  modified  from  the  present  set  or  with 
completely  new  ones. 

Approach 

The  present  analysis  is  based  on  proposals  to  exploit  learning  science  opportunities 
related  to  TRADOC ’s  mission.  The  proposals  were  derived  from  ideas  put  forth  during  the  Army 
Science  of  Learning  Workshop.  Each  proposal  was  examined  for  its  benefits  and  costs. 

To  maximize  the  benefits  from  the  proposals  without  regard  to  costs,  the  optimal  plan 
would  be  to  select  all  proposals.  However,  such  costs  would  be  prohibitively  expensive.  On  the 
other  hand,  to  minimize  costs  without  regard  to  benefit,  the  optimal  plan  would  be  to  select  the 
least  expensive  proposal  (or  set  of  proposals).  However,  this  approach  provides  no  assurance  that 
the  results  would  be  of  value  to  the  Army. 

The  present  analytic  approach  represents  a  more  balanced  strategy  that  casts  costs  and  benefits  in 
a  trade  space  from  which  several  alternative  plans  can  be  derived  and  compared.  The  intent  of 
this  section  is  to  document  this  approach  in  sufficient  detail  to  allow  it  to  be  replicated  with 
alternate  proposals  and/or  a  new  set  of  estimated  values. 

Proposals 

The  elements  of  the  current  analysis  are  21  proposed  R&D  efforts.  The  proposals  were 
compiled  by  the  Institute  for  Defense  Analyses  (IDA)  based  on  some  of  the  more  important 
concepts  that  evolved  during  the  workshop’s  sessions.  These  proposals  were  not  always 
discussed  directly  during  the  workshop;  rather,  they  were  implied  by  ideas  that  the  workshop’s 
participants  generated. 

Each  proposal  was  given  a  long  title  to  describe  its  content  and  a  short  title  (derived  from 
the  longer  version)  to  facilitate  referring  to  these  proposals  in  the  analyses.  Table  1  summarizes 
the  proposals  by  providing  the  long  and  short  versions  of  the  titles,  the  working  group  that 
generated  the  proposal,  and  a  statement  of  purpose. 
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Table  1 

Summaries  of  Research  Proposals 


Long  Title  Short  Title  Statement  of  Purpose 


Learning  Model  Working  Group 


Selective  Implementation  of 
the  Guided  Experiential 
Learning  (GEL)  Model 

The  GEL  Model 

Implementation  of  a  Human 
Performance  Improvement 
(HPI)  Program  in  the  Army 

HPI  in  the  Army 

Blending  Distance  Learning 
With  Traditional  Instruction 

Blended  Learning 

Models  To  Predict 

Effectiveness  and  Efficiency  of 
Training  and  Education 

Predictive  Models 

Effective  and  Efficient 

Knowledge 

Knowledge  Acquisition, 
Management,  and  Sharing 
Approaches  and  Tools 

Management 

Life-Long  Learning  Models 

With  Intelligent  Agents  for 
Individualized  Guidance 

Life-Long  Learning 

Apply  a  GEL  model  to  a  small  but  diverse  set  of  training 
objectives 

Study  HPI  to  determine  the  appropriate  method  for 
implementing  it  in  the  Army 

Review  civilian  literature  on  blending  virtual  and  live 
training  to  interpret  application  to  constraints  of  Army 
training 

Based  on  modem  learning  science,  develop  models  to 
predict  the  effectiveness  and  efficiency  of  training  and 
education 

Develop  an  appropriate  architecture  for  knowledge 
management  and  supporting  tools  for  the  Army 


Examine  alternative  models  for  using  intelligent  agents  to 
perform  mentoring  functions 


Train  Soldiers  Working  Group 


Systematic  Implementation 

Plan  for  Distributed  Learning 
(dL) 

Implementation 
Plan  for  dL 

Research  and  Development  on 
Distributed  Learning  (dL) 

Topics 

R&D  on  dL 

Analyses  of  Training 

Analyses  of 
Training 

Implementation  of  Enhanced 
Training  Capabilities 

Training 

Capabilities 

Guidelines  for  Tools  and 
Methods  To  Support  Training 

Training  Support 
Tools 

New  Programs  of  Research  in 
Training 

New  Training  R&D 

Existing  and  Evolving 

Strategies  for  Tailoring 

Training 

Tailored  Training 

Use  dL  technology  to  accelerate  training,  reduce  costs, 
reduce  personnel  requirements,  and  improve  operational 
effectiveness 

Encourage  the  establishment  of  R&D  programs  to  guide 
the  development  and  selection  of  effective  dL  strategies, 
technologies,  and  authoring  tools 

Perform  analyses  of  training  inputs  and  outputs  to 
accelerate  training  and  reduce  training  costs  and 
personnel  requirements 

Develop  capabilities  related  to  Assignment-Oriented 
Training  (AOT)  and  for  training  in  units 

Reduce  the  “demand  side"  of  training  through  increased 
use  of  performance  and  decision  aiding  technology 
integrated  with  training  design  and  objectives 

Encourage  R&D  programs  to  develop  a  full  systems 
approach  that  considers  costs  and  benefits  of  all 
alternatives  for  producing  human  performance 

Systematically  apply  proven  techniques  for  tailoring 
training  to  needs,  prior  knowledge,  capabilities,  and 
interests  of  individual  Soldiers 

(Continued) 
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Table  1 — Continued 


Long  Title 

Short  Title 

Statement  of  Purpose 

Train  Soldiers  Working  Group  (Continued) 

Existing  and  Evolving 

Strategies  for  Real-Time 
Diagnostic  Evaluation 

Diagnostic 

Evaluation 

Exploit  and  apply  proven  capabilities  related  to  real-time, 
diagnostic  assessment  performed  using  both  explicit 
testing  and  routine  human-computer  interactions 

Rapid  Conversion  of  Lessons 
Learned 

Lessons  Learned 

Develop  and  improve  capabilities  to  rapidly  collect, 
understand,  and  disseminate  lessons  learned  in  theater 

Develop  Leaders  Working  Group 

Leader  Socialization 

Leader 

Socialization 

Investigate  collaborative  technologies  for  influencing 
socialization,  particularly  in  combination  with  the  Army 

Force  Generation  (ARFORGEN)  model 

Critical  Thinking  Skills 

Critical  Thinking 
Skills 

Produce  courses  that  blend  previous  experience  in 
teaching  critical  thinking  with  specific  recommendations  to 
improve  the  training  and  use  of  such  skills 

Experience-Based  Learning 
Methods 

Experience-Based 

Learning 

Develop  multimedia  technology  that  encourages  leaders 
to  discover  and  learn  about  past  conflicts  from  different 
perspectives  and  analytical  approaches 

Future  Capabilities  Working  Group 

Improvisation 

Improvisation 

Directly  teach  the  art  of  improvisation  to  address  the  need 
for  “adaptive"  capabilities 

Folksonomies 

Folksonomies 

Develop  tools  that  would  allow  individuals  and  teams  of 
deployed  personnel  to  document  and  organize  their 
experiences 

Team  Task  Analysis/Behavior 
Modeling 

Team  Task 

Analysis 

Develop  tools  to  analyze  task  performance  from 
anthropological  models 

Appendix  A  provides  a  detailed  description  of  the  proposals.  Each  description  includes 
an  explanation  of  the  underlying  concept  of  the  effort  and  an  estimate  of  required  time  and 
personnel  resources.  The  order  of  proposals  in  Table  1  is  the  same  order  used  Appendix  A,  but 
this  order  does  not  reflect  priority  in  cost  or  benefit. 

Cost  Estimation 

For  each  proposal,  the  individual  who  developed  the  concept  also  provided  cost 
estimates.  These  estimates  were  decomposed  into  the  following  cost  elements: 

•  Implementation  costs.  These  are  the  costs  required  to  develop  and  implement  the 
intervention,  tool,  or  methods  and  are  estimated  by  the  number  of  professional  staff- 
months  required. 
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•  Maintenance  costs.  These  are  the  costs  required  to  maintain  the  intervention,  tool,  or 
methods.  The  cost  metric  was  the  number  of  staff-months  per  year. 

•  Other  costs.  T  These  are  the  significant  cost  elements  required  to  implement  a 
proposal  above  and  beyond  personnel  costs.  Examples  include  specific  computer 
hardware  or  software. 

Total  costs  were  calculated  in  two  ways.  First,  the  “implementation”  and  “other”  costs 
were  summed  to  estimate  the  first-year  (Yl)  start-up  costs.  Second,  a  more  long-term  estimate 
was  calculated  by  adding  “maintenance”  costs  to  Yl  costs,  assuming  a  5-year  time  frame  to 
correspond  with  standard  Future  Years  Defense  Plan  (FYDP)  estimates. 

Benefit  Estimation 

For  the  present  purposes,  the  benefit  of  a  proposed  R&D  effort  was  measured  by  its 
“expected  impact.”  This  value  was  conceived  as  being  analogous  to  expected  value  and  was 
defined  as  the  product  of  the  work’s  operational  impact  and  the  probability  of  success.  The  two 
component  values  of  benefits — impact  and  probability  of  success — were  put  into  operation  as 
follows. 

Impact.  The  method  for  estimating  impact  was  adopted  from  the  Simple  Multiattribute 
Rating  Technique  (SMART)  developed  by  Edwards  (1977).  Initially,  each  of  the  proposal 
writers  independently  ranked  the  entire  set  of  proposals  from  1  (most  impact)  to  21  (least 
impact) — with  ties  in  ranks  allowed.  After  the  estimates  were  recorded,  the  writers  then  met 
face-to-face  to  resolve  differences  in  the  proposals’  rank  ordering.  Given  the  rank  orders,  the 
raters  were  tasked  to  convert  the  ranks  to  ratio  measures  by  assigning  the  least  important 
proposal  an  arbitrary  value  of  10.1  The  rater  then  considered  the  next  least  important  proposal 
and  determined  how  much  more  important  (if  at  all)  it  was  than  the  least  important.  This 
proposal  then  was  assigned  a  number  reflecting  that  ratio.  This  process  was  continued  up  the  list, 
checking  each  set  of  implied  ratios  as  each  new  judgment  was  made.  Thus,  if  one  proposal  was 
assigned  a  weight  of  20  and  another  was  assigned  a  weight  of  80,  it  meant  that  the  20  proposal 
was  one- fourth  as  important  as  the  80  proposal,  and  so  forth.  The  upper  ends  of  the  scale  were 
allowed  to  vary,  resulting  in  scales  ranging  from  10—40  to  10-200.  The  final  scale  was  a 
weighted  average  of  the  individual  rater  scales,  which  were  transformed  so  that  the  range  of 
estimated  proposal  impact  varied  from  10  (least  important)  to  100  (most  important). 

Probability  of  success.  Along  with  rating  the  importance  of  the  proposals,  the  proposal 
writers  estimated  the  probability  of  success,  principally  as  a  function  of  the  proposal’s  technical 
feasibility,  although  some  consideration  was  given  to  the  practical  constraints  of  executing  the 
research,  such  as  organizational  missions,  policies,  and  traditions  that  may  change  only  with 
intervention  at  the  level  of  the  Army  Chief  of  Staff.  Like  the  impact  estimates,  raters 
independently  estimated  these  probabilities. 


1  According  to  Edwards  ( 1 977),  the  starting  point  1 0  “. . .  is  used  rather  than  1  to  permit  subsequent 

judgments  to  be  finely  graded  and  nevertheless  made  in  integers”  (p.  328). 
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Compilation/Presentation  of  Cost-Benefit  Estimations 

These  initial  estimates  were  compiled  and  presented  to  all  the  proposal  writers  in  a  face- 
to-face  meeting.  The  resulting  values  are  averages  that  the  proposal  writers  agreed  captured  the 
group’s  consensus. 

Results 

This  section  presents  results  from  the  analyses  of  the  21  R&D  opportunities  derived  from 
the  Army  Science  of  Learning  Workshop.  These  results  refer  to  the  proposals  by  short  titles. 
Table  1  lists  the  corresponding  longer  titles,  and  Appendix  A  contains  a  description  of  the 
proposals. 

Cost  Analysis 

Table  2  summarizes  the  estimated  costs  related  to  the  21  proposed  efforts.  For  the  staff 
estimates,  note  that  most  efforts  allowed  for  parallel  work  from  multiple  staff  members  so  that 
the  chronological  time  to  develop  was  often  much  less  than  the  staffing  estimates.  The  Y 1  start¬ 
up  costs  (i.e.,  development  and  implementation)  ranged  from  S187.5K.  to  $9.4M  for  individual 
efforts,  with  a  total  of  S23.4M  for  the  entire  set  of  21  proposals.  The  total  estimated  costs  over  a 
5-year  FYDP  planning  cycle  showed  more  variability,  ranging  from  S187.5K  to  S46.9M,  with  a 
total  of  $76.4M  for  the  entire  set.  These  total  potential  costs  varied  more  than  the  start-up  costs 
because  they  included  the  start-up  costs  plus  the  costs  of  maintaining  tools  and  methods  and/or 
additional  related  costs. 

Benefits  Analysis 

Table  3  presents  the  two  components  of  the  benefits  analysis:  the  multiplicative  product 
of  the  probability  of  success,  p( S),  and  its  rated  impact.  The  values  of  p( S)  were  generally 
moderate  to  high,  ranging  from  .45  to  .85,  which  reflected  the  analysts’  determination  that  most 
of  the  proposed  efforts  were  technically  feasible  (i.e.,  not  highly  risky).  The  impact  ratings 
ranged  from  10  to  100  and  represented  the  numerical  ratios  among  proposals.  For  instance,  the 
highest  rated  proposal  was  estimated  to  have  10  times  the  impact  of  the  lowest  rated  proposal. 
The  two  ratings  were  positively — but  not  significantly — correlated  (r  =  .35,  p  >  .  1 1 ),  suggesting 
that  the  judges  were  rating  the  proposals  on  substantially  different  dimensions. 

The  benefit  value  for  each  proposal  was  calculated  and  then  summed  over  the  entire  set 
of  2 1  proposals.  The  total  was  used  to  determine  the  proportion  of  total  benefit  attributed  to  each 
proposed  effort,  as  shown  in  the  right-most  column  in  Table  3.  These  values  for  the  individual 
proposed  efforts  ranged  from  almost  1%  to  over  13%  of  the  total  benefit  of  the  entire  set  of 
proposals. 
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Table  2 


Costs  Related  to  Proposals 


Short  Titles 

Personnel  Costs 
(Staff-Months) 

Total  Costs 
(K  Dollars) 

Implementation 

Maintenance 
(per  year) 

Y1 

Costs 

FYDP 

Costs 

The  GEL  Model 

12 

6 

250 

750 

HPI  in  the  Army 

24 

0 

500 

500 

Blended  Learning 

9 

0 

188 

188 

Predictive  Models 

36 

0 

750 

750 

Knowledge  Management 

48 

12 

1,000 

2,000 

Life-Long  Learning 

48 

12 

1,000 

2,000 

Implementation  Plan  for  dL 

34 

2 

708 

875 

R&D  on  dL 

24 

24 

500 

2,500 

Analyses  of  Training 

14 

8 

292 

958 

Training  Capabilities 

18 

3 

375 

625 

Training  Support  Tools 

48 

6 

1,000 

1,500 

New  Training  R&D 

18 

18 

9,375a 

46,875a 

Tailored  Training 

12 

3 

250 

500 

Diagnostic  Evaluation 

36 

12 

750 

1,750 

Lessons  Learned 

24 

12 

500 

1,500 

Leader  Socialization 

26b 

0 

542 

542 

Critical  Thinking  Skills 

60 

4 

1,250 

1,583 

Experience-Based  Learning 

24 

1 

500 

583 

Improvisation 

21c 

9d 

438 

1,188 

Folksonomies 

36 

0 

2,250e 

8,250e 

Team  Task  Analysis 

48 

0 

1,000 

1,000 

Note.  Personnel  costs  based  on  $20,833  per  staff-month  (or  $250,000  per  staff-year).  Also,  costs  are 
rounded  to  the  nearest  thousand. 

aln  addition  to  staffing  costs,  totals  include  $9M/year  budget  for  extramural  research. 
blncludes  additional  personnel  costs  required  to  develop  a  single  test  course, 
includes  additional  personnel  costs  related  to  curriculum  development  in  first  year, 
includes  personnel  costs  related  to  follow-on  assessment. 
elncludes  game  development  and  hardware  costs. 
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Table  3 


Benefits  Related  to  Proposals 


Short  Titles 

P(S) 

Impact 

Rating 

(10-100) 

Proportion 

Total 

Benefit 

The  GEL  Model 

0.75 

15 

0.0177 

HPI  in  the  Army 

0.45 

10 

0.0071 

Blended  Learning 

0.80 

15 

0.0189 

Predictive  Models 

0.50 

45 

0.0355 

Knowledge  Management 

0.70 

75 

0.0828 

Life-Long  Learning 

0.60 

55 

0.0520 

Implementation  Plan  for  dL 

0.85 

100 

0.1340 

R&D  on  dL 

0.65 

40 

0.0410 

Analyses  of  Training 

0.65 

65 

0.0666 

Training  Capabilities 

0.80 

35 

0.0441 

Training  Support  Tools 

0.75 

20 

0.0236 

New  Training  R&D 

0.50 

15 

0.0118 

Tailored  Training 

0.80 

90 

0.1135 

Diagnostic  Evaluation 

0.70 

85 

0.0938 

Lessons  Learned 

0.70 

60 

0.0662 

Leader  Socialization 

0.65 

20 

0.0205 

Critical  Thinking  Skills 

0.50 

55 

0.0434 

Experience-Based  Learning 

0.75 

40 

0.0473 

Improvisation 

0.55 

15 

0.0130 

Folksonomies 

0.65 

10 

0.0102 

Team  Task  Analysis 

0.80 

45 

0.0568 

Cost-Benefit  Tradeoffs 

Table  4  summarizes  the  values  used  to  determine  cost-benefit  tradeoffs.  For  each 
proposal,  the  benefit-to-cost  ratio  (i.e.,  benefit  per  unit  cost)  was  calculated  twice:  for  Y1  costs 
and  for  FYDP  costs.  These  ratios  were  then  used  to  rank  the  proposed  efforts  from  1  (the  most 
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Table  4 

Benefit-to-Cost  Ratios 


Short  Titles 

Y1  Costs 

FYDP  Costs 

Benefit-to-Cost 

Ratio 

Rank 

Benefit-to-Cost 

Ratio 

Rank 

The  GEL  Model 

0.0450 

11 

0.0150 

15 

HPI  in  the  Army 

0.0090 

19 

0.0090 

18 

Blended  Learning 

0.0640 

7 

0.0640 

3 

Predictive  Models 

0.0300 

14 

0.0300 

9 

Knowledge  Management 

0.0525 

9 

0.0263 

11 

Life-Long  Learning 

0.0330 

13 

0.0165 

14 

Implementation  Plan  for  dL 

0.1200 

3 

0.0971 

2 

R&D  on  dL 

0.0520 

10 

0.0104 

16 

Analyses  of  Training 

0.1449 

2 

0.0441 

6 

Training  Capabilities 

0.0747 

6 

0.0448 

5 

Training  Support  Tools 

0.0150 

18 

0.0100 

17 

New  Training  R&D 

0.0008 

21 

0.0002 

21 

Tailored  Training 

0.2880 

1 

0.1440 

1 

Diagnostic  Evaluation 

0.0793 

5 

0.0340 

8 

Lessons  Learned 

0.0840 

4 

0.0280 

10 

Leader  Socialization 

0.0240 

15 

0.0240 

12 

Critical  Thinking  Skills 

0.0220 

16 

0.0174 

13 

Experience-Based  Learning 

0.0600 

8 

0.0514 

4 

Improvisation 

0.0189 

17 

0.0069 

19 

Folksonomies 

0.0029 

20 

0.0008 

20 

Team  Task  Analysis 

0.0360 

12 

0.0360 

7 
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cost  beneficial)  to  21  (least  cost  beneficial).  The  effect  of  calculating  benefit-to-cost  ratios  based 
on  FYDP  vs.  Y1  costs  was  to  change  the  relative  rankings  of  the  proposed  efforts.  Overall,  the 
rankings  differed  by  4  or  more  positions  for  8  of  the  21  proposals.  Specifically,  relatively  large 
maintenance  costs  increased  total  FYDP  costs  and  reduced  the  benefit-to-cost  rankings  of  four 
proposals:  (1)  The  GEL  Model,  (2)  R&D  on  dL,  (3)  Analyses  of  Training,  and  (4)  Lessons 
Learned.  Likewise,  the  lack  or  relative  lack  of  maintenance  costs  reduced  total  FYDP  costs  and 
increased  the  relative  standings  of  four  other  proposals:  (1)  Blended  Learning,  (2)  Predictive 
Models,  (3)  Experience-Based  Learning,  and  (4)  Team  Tasks  Analysis. 

These  benefit-to-cost  ratios  were  then  used  to  select  optimal  subsets  or  “packages”  of 
proposed  efforts.  To  illustrate  this  process,  consider  a  package  that  contains  a  single  proposal. 
The  optimal  selection  would  be  the  proposal  with  the  greatest  benefit-to-cost  ratio.  The  optimal 
two-proposal  package  would  then  include  that  same  proposal  plus  the  one  with  the  next  highest 
benefit-to-cost  ratio.  By  ordering  the  proposals  in  this  way,  one  can  calculate  the  total  benefit  of 
the  proposal  packages  as  a  function  of  their  cumulative  costs.  More  generally,  this  function 
reveals  packages  of  proposals  that  optimize  specific  benefit  and  cost  levels. 

Tradeoff's  using  Y1  (Start-up)  Costs.  To  illustrate  this  concept,  Figure  1  displays  the 
cumulative  Y1  costs  and  benefits  of  proposed  efforts  ordered  by  benefit-to-cost  ratios.  The  slope 
starts  out  very  steep  but  decreases  over  the  range  of  this  curvilinear  function.  Also,  as  noted  on 
the  graph,  the  last  proposal  (New  Training  R&D)  added  very  little  additional  benefit,  considering 
its  relatively  large  costs.  Consequently,  this  proposal  was  an  outlier  relative  to  the  20  other 
proposals  and  could  not  be  shown  in  its  actual  position  without  distorting  the  function. 

(Table  B-l  in  Appendix  B  displays  the  cumulative  Y1  costs  and  benefits  for  all  proposals.)  The 
following  discussion  shows  that  the  method  is  sensitive  to  key  aspects  of  cost-benefit  analysis 
(i.e.,  maximize  benefit/minimize  cost,  fixed  costs,  fixed  benefits). 

1 .  Maximize  benefits/minimize  costs.  There  is  a  point  in  the  function  where  the  returns  in 
benefit  begin  to  diminish  rapidly  for  cumulative  investments.  This  point  appears  to  be 
at  about  the  fifth  proposal  (see  Figure  1).  Consequently,  a  package  comprising  those 
five  proposals  (Tailored  Training,  Analyses  of  Training,  Implementation  Plan  for  dL, 
Lessons  Learned,  and  Diagnostic  Evaluation)  would  provide  nearly  47%  of  the  total 
benefit  of  the  entire  set  of  proposals  for  less  than  1 1%  of  the  total  costs  (or  S2.5M  in 
Y1  costs). 

2.  Fixed  costs.  In  this  case,  the  funding  organization  has  a  fixed  amount  of  money  for 
investment  and  wants  to  ensure  that  it  constructs  the  optimal  package  (i.e.,  the  one 
that  maximizes  total  benefit).  The  procedure  is  to  choose  the  proposals  in  order  of 
cost-benefit  ratio  until  the  total  cumulative  cost  is  approximately  equal  to  the  fixed 
amount.  Using  the  present  data  as  an  example,  suppose  that  a  total  of  $5M  was 
available  in  Y1  money.  Assuming  that  the  total  must  not  exceed  this  budget,  the 
optimal  package  would  then  comprise  the  first  five  proposals  in  the  previous  package 
plus  four  more:  Training  Capabilities,  Blended  Learning,  Experience-Based  Learning, 
and  Knowledge  Management.  This  nine-proposal  package  would  account  for  over 
two-thirds  (67%)  of  the  total  benefit. 
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Figure  1.  Incremental  benefit  as  a  function  of  cumulative  Y1  costs. 

3.  Fixed  costs.  In  this  case,  the  funding  organization  has  a  fixed  amount  of  money  for 
investment  and  wants  to  ensure  that  it  constructs  the  optimal  package  (i.e.,  the  one 
that  maximizes  total  benefit).  The  procedure  is  to  choose  the  proposals  in  order  of 
cost-benefit  ratio  until  the  total  cumulative  cost  is  approximately  equal  to  the  fixed 
amount.  Using  the  present  data  as  an  example,  suppose  that  a  total  of  $5M  was 
available  in  Y1  money.  Assuming  that  the  total  must  not  exceed  this  budget,  the 
optimal  package  would  then  comprise  the  first  five  proposals  in  the  previous  package 
plus  four  more:  Training  Capabilities,  Blended  Learning,  Experience-Based  Learning, 
and  Knowledge  Management.  This  nine-proposal  package  would  account  for  over 
two-thirds  (67%)  of  the  total  benefit. 


4.  Fixed  benefits.  In  this  case,  the  funding  organization  wants  to  ensure  that  a  package  of 
proposals  provides  some  stated  level  of  benefit.  The  optimal  package  in  this  case  is 
the  one  that  minimizes  cost  for  the  fixed  benefit.  For  the  sake  of  illustration,  suppose 
the  funding  organization  wanted  a  package  that  accounted  for  at  least  80%  of  the 
benefit  derived  from  the  total  set  of  21  proposals.  The  optimal  package  (i.e.,  the  one 
that  minimizes  Y1  cost)  would  then  comprise  the  nine  proposals  specified  in  the 
previous  package  plus  four  more:  R&D  on  dL,  The  GEL  Model,  Team  Task  Analysis, 
and  Life-Long  Learning.  This  13-proposal  package,  which  accounts  for 
approximately  83%  of  the  benefit,  would  cost  $7.3M,  which  represents  about  31%  of 
the  total  cost  of  all  21  proposals. 
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Tradeoffs  using  FYDP  costs.  Figure  2  presents  the  same  sort  of  information  as  Figure  1 , 
but  this  information  is  based  on  FYDP  costs  instead  of  Y1  costs.  As  discussed  previously,  this 
distinction  changed  the  rank  order  of  the  proposals  (i.e.,  considering  FYDP  costs  instead  of  Y1 
costs  provides  different  optimal  selections).  As  in  Figure  1,  the  New  Training  R&D  proposal  was 
the  last  to  be  added  and  is  not  shown  in  its  actual  place  on  the  function.  (Table  B-2  in 
Appendix  B  displays  the  cumulative  FYDP  costs  and  benefits  for  all  proposals.)  The  examples 
that  follow  are  meant  to  parallel  those  described  for  Y1  costs. 


Figure  2.  Incremental  benefit  as  a  function  of  cumulative  FYDP  costs. 

1 .  Maximize  benefits/minimize  costs.  The  point  where  the  returns  in  benefit  begin  to 
diminish  rapidly  for  cumulative  investments)  is  not  as  obvious  in  Figure  2  as  it  is  in 
Figure  1.  However,  to  parallel  the  previous  analysis,  the  first  five  proposals  were 
chosen:  Tailored  Training,  Implementation  Plan  for  dL,  Blended  Learning, 
Experience-Based  Learning,  and  Training  Capabilities.  This  package  appears  cost- 
beneficial  in  that  well  over  one-third  of  the  total  benefit  (36%)  would  cost  about 
$2.8M.  However,  this  cost  is  a  much  smaller  fraction  of  the  total  cost  (about  4%) 
because  the  FYDP  costs  have  a  much  larger  range  than  the  Y1  costs.  Also,  although 
this  five-proposal  FYDP  package  roughly  parallels  the  corresponding  Y1  package, 
the  contents  are  substantially  different.  Of  the  five  proposals  in  both  the  Y1  and 
FYDP  analyses,  only  two  (Tailored  Training  and  Implementation  Plan  for  dL)  are 
common  to  both  lists. 

* 

2.  Fixed  costs.  It  would  seem  reasonable  that  the  amount  of  FYDP  money  available 
would  be  greater  than  the  amount  of  Y1  money  available.  Suppose  that  $8M  was 
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available  across  the  FYDP,  which  provided  the  cap  for  the  budget.  In  this  case,  the 
optimal  package  (i.e.,  the  one  that  maximizes  benefit)  would  also  comprise  nine 
proposals — the  five  contained  in  the  previous  package  plus  four  more:  Analyses  of 
Training,  Team  Tasks  Analysis,  Diagnostic  Evaluation,  and  Predictive  Models.  This 
package  would  consume  $7.2M  of  the  $8M  budget  and  would  account  for  the 
maximum  benefit  possible  (61%  of  the  total). 

3.  Fixed  benefits.  Paralleling  the  Y1  package,  it  is  assumed  that  the  funder  desires  a 
package  that  accounts  for  at  least  80%  of  the  total  benefit.  The  optimal  package  (i.e., 
the  one  that  minimizes  cost)  would  then  comprise  the  nine  proposals  specified  in  the 
previous  package  plus  four  more  (Lessons  Learned,  Knowledge  Management,  Leader 
Socialization,  and  Critical  Thinking  Skills).  This  13-proposal  package,  which 
accounts  for  approximately  82%  of  the  benefit,  would  cost  about  $12.9M. 

Summary  of  Packages 

Table  5  summarizes  these  various  packages  of  proposals  derived  from  Y1  and  FYDP 
costs.  The  proposal  numbers  (e.g.,  1, 2, 3,  and  so  forth)  refer  to  their  relative  rank  in  benefit-to- 
cost  ratios,  starting  with  the  most  cost-beneficial  proposal.  For  packages  that  have  similar 
features  (rows  in  the  table),  the  results  show  differences  because  of  cost  definition  (Y1  or 
FYDP).  As  the  packages  decrease  in  cost-benefit  (see  the  bottom  row  in  particular),  the  two 
definitions  produce  greater  divergence.  Using  the  Y1  definition,  the  bottom  row  includes  three 
proposals  that  do  not  appear  on  any  selections  using  the  FYDP  definition:  R&D  on  dL,  The  GEL 
Model,  and  Life-Long  Learning.  These  results  suggest  that  these  three  proposals  accrue  a  short¬ 
term  cost-benefit  that  is  not  reflected  in  the  FYDP  cost  definition.  Likewise,  using  the  FYDP 
definition,  the  bottom  row  contains  two  proposals  that  do  not  appear  on  any  of  the  selections 
using  the  Y1  definition:  Leader  Socialization  and  Critical  Thinking  Skills.  These  results  suggest 
that  these  two  proposals  are  viewed  as  cost  effective  only  if  a  longer  costing  window  (5  years)  is 
assumed. 

The  observed  differences  discussed  previously  demonstrate  that  the  methods  described 
herein  (Y1  and  FYDP)  are  sensitive  to  the  costing  method  used.2  Despite  these  differences,  some 
commonality  existed  in  proposal  selections  between  the  two  methods.  To  provide  a  single 
recommendation  that  balances  short-  and  long-term  perspectives,  a  “compromise”  package  was 
assembled.  This  package  comprised  the  top  10  proposals  based  on  the  average  rank  for  the  two 
benefit-to-cost  ratios.  Table  6  lists  the  resulting  package  in  priority  order,  from  most  to  least 
cost-benefit.  As  indicated,  the  total  cost  of  this  package  over  the  FYDP  would  be  less  than 
$10M.  However,  over  half  of  that  FYDP  costs  ($5.6M)  would  be  incurred  in  Year  1. 


There  is  reason  to  believe  that  the  method  would  be  similarly  sensitive  to  the  method  for  determining 
benefit,  although  no  alternative  was  considered  for  the  present  set  of  proposals. 


13 


Table  5 


Optimal  Proposal  Packages  Based  on  Y1  versus  FYDP  Cost  Estimates 


Based  on  Y1  Costs 

Based  on  FYDP  Costs 

Package 

Feature 

Proposals 

Cost 

Percent 

Total  Benefit 

Proposals 

Cost 

Percent 

Total  Benefit 

Maximize 

benefits/ 

minimize 

1.  Tailored  Training 

2.  Analyses  of  Training 

1.  Tailored  Training 

2.  Implementation  Plan  for  dL 

3.  Implementation  Plan  for  dL 

S2.5M 

47% 

3.  Blended  Learning 

$2.8M 

36% 

cost 

4.  Lessons  Learned 

4.  Experience-Based  Learning 

5.  Diagnostic  Evaluation 

5.  Training  Capabilities 

All  above  (Y1),  plus 

All  above  (FYDP),  plus 

Fixed 

costs 

6.  Training  Capabilities 

6.  Analyses  of  Training 

7.  Blended  Learning 

S4.6M 

67% 

7.  Team  Task  Analysis 

$7.2M 

61% 

8.  Experience-Based  Learning 

8.  Diagnostic  Evaluation 

9.  Knowledge  Management 

9.  Predictive  Models 

All  above  (Y1),  plus 

All  above  (FYDP),  plus 

Fixed 

benefits 

10.  R&D  on  dL 

11.  The  GEL  Model 

$7.3M 

83% 

10.  Lessons  Learned 

11.  Knowledge  Management 

$12. 9M 

82% 

12.  Team  Task  Analysis 

12.  Leader  Socialization 

13.  Life-Long  Learning 

13.  Critical  Thinking  Skills 

Illustration  of  Selection  Outcomes 

Ultimately,  the  validity  of  the  method  is  a  function  of  the  benefit-to-cost  ratios.  The 
discussion  that  follows  documents  how  these  ratios  are  determined  and  how  the  values  determine 
the  selection  of  specific  proposals.  The  intent  is  to  help  the  reader  understand  current  results  and 
also  to  understand  the  method’s  use  for  future  applications.  Nine  proposals  were  chosen  either 
because  they  consistently  ranked  high  or  low  in  benefit-to-cost  ratios  or  their  rank  was 
substantially  changed  because  of  the  cost  definition  used  (Y1  vs.  FYDP). 

Proposals  consistently  ranked  high.  Two  proposals  (Tailored  Training  and 
Implementation  Plan  for  dL)  were  consistently  ranked  high  in  benefit-to-cost  ratios,  regardless  of 
cost  definitions.  The  two  proposals  were  similar  because  the  same  three  factors  accounted  for 
high  rank: 

1 .  Both  proposals  were  based  on  a  proven  record  of  research;  thus,  the  probability  of 
success,  p(S),  was  rated  high:  .80  for  Tailored  Training  and  .85  for  Implementation 
Plan  for  dL. 
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Table  6 

Recommended  Package  of  Proposals  Based  on  Both  Cost  Estimates 


Proposals 

Y1  Cost  ($K) 

FYDP  Cost  ($K) 

1. 

Tailored  Training 

250 

500 

2. 

Implementation  Plan  for  dL 

708 

875 

3. 

Analyses  of  Training 

292 

958 

4. 

Blended  Learning 

188 

188 

5. 

Training  Capabilities 

375 

625 

6. 

Experience-Based  Learning 

500 

583 

7. 

Diagnostic  Evaluation 

750 

1,750 

8. 

Lessons  Learned 

500 

1,500 

9. 

Team  Task  Analysis 

1,000 

1,000 

10. 

Knowledge  Management 

1,000 

2,000 

Totals 

5,563 

9,979 

2.  Both  proposals  were  considered  highly  relevant  to  current  training  operations,  and 
were  thus  rated  high  for  their  potential  impact.  In  fact,  these  two  proposals  received 
the  two  highest  impact  ratings  out  of  all  21  proposals:  100  for  Implementation  Plan 
for  dL  and  90  for  Tailored  Training. 

3.  Both  proposals  incurred  some  maintenance  costs,  but  these  costs  were  relatively  low 
(i.e.,  a  fraction  of  a  staff-year).  Their  total  FYDP  costs,  while  not  the  lowest  of  the 
entire  set,  were  relatively  low. 

Proposals  consistently  ranked  low.  Three  proposals  (New  Training  R&D,  Folksonomies, 
and  Implementation  of  HPI)  were  similar  because  they  were  consistently  ranked  low  in  benefit- 
to-cost  ratios.  However,  the  reasons  for  this  result  were  different  for  the  three  proposals.  These 
findings  were  particularly  useful  because  they  revealed  conditions  under  which  proposals  are  not 
selected. 

1 .  The  New  Training  R&D  proposal  was  rated  low  in  impact  (1 5  on  a  scale  of  1 00)  and 
low  in  probability  of  success  (.5).  In  the  defense  of  these  low  ratings,  we  believe  that 
good  research  should  be  technically  challenging  (i.e.,  low  in  probability  of  success) 
and  not  necessarily  applicable  to  operation — at  least  immediately.  However,  our 
assumption  was  that  TRADOC  was  more  interested  in  applied  (as  opposed  to 
theoretical)  research,  and  the  definitions  of  impact  and  probability  of  success  reflect 
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that  assumption.  The  New  Training  R&D  proposal  was  also  the  most  costly  of  the 
21  proposals,  incurring  nearly  $47M  over  the  FYDP.  In  addition  to  a  small  staffing 
requirement,  this  total  includes  the  cost  of  the  research  itself,  estimated  to  be  $9M  per 
year.  Obviously,  the  cost  would  be  drastically  reduced  for  the  funding  organization  if 
research  funds  were  provided  by  an  external  organization. 

2.  The  Folksonomies  proposal  was  based  on  established  anthropological  methodology, 
so  the  value  for  p( S)  was  rated  as  fairly  moderate  (.65).  Two  other  factors  were  more 
important  to  its  consistent  low  rating:  First,  the  overall  FYDP  costs  (S8.25M)  were 
second  only  to  New  Training  R&D,  largely  because  these  costs  included  $7.5M  for 
hardware  and  for  software  development.  If  this  proposal  were  reconsidered,  the 
validity  of  these  cost  estimates  would  be  a  likely  target.  Second,  the  Folksonomies 
proposal  (along  with  Implementation  of  HPI)  shared  the  lowest  impact  rating  (10)  of 
all  proposals.  It  was  based  on  anthropological  methods  with  which  the  raters  were 
less  than  fully  conversant.  For  this  proposal  to  be  reconsidered,  increased  emphasis 
would  have  to  be  placed  on  methodology  and  the  description  of  potential  impact. 

3.  Implementation  of  HPI.  Ironically,  the  proposal  writers  thought  that  the  potential 
impact  of  the  HPI  concept  (if  implemented  correctly)  was  one  of  the  largest  in  the  set. 
The  problem  is  that  the  large  potential  impact  came  at  a  high  cost — in  terms  of 
financial  investment  and  disruption  to  the  organization.  To  limit  cost  risks,  the 
proposal  was  scoped  back  to  a  simple  assessment  of  alternatives  for  planning 
purposes.  Although  this  approach  clearly  limited  the  investment  (S500K.  over  the 
entire  FYDP),  it  also  limited  the  estimated  impact  of  the  proposal  to  one  of  the  lowest 
values  in  the  set.  Furthermore,  even  though  HPI  has  worked  well  for  the  Navy  and  the 
Coast  Guard,  the  raters  were  skeptical  that  it  could  be  implemented  as  well  in  the 
Army.  Thus,  the  probability  of  success  (.45)  was  also  markedly  low  for  this  proposal. 
An  unintentional  finding  from  this  proposal  is  that  the  method  is  very  sensitive  to  the 
scope  of  the  proposed  effort. 

Proposals  affected  by  cost  definitions.  The  rankings  of  four  proposals  were  especially 
sensitive  to  cost  definitions,  either  declining  or  increasing  when  total  FYDP  costs  were 
considered. 

1.  Proposals  Rated  Lower  Using  FYDP  vs.  Y1  Cost  Definitions.  Two  proposals  (R&D 
on  dL  and  Lessons  Learned)  substantially  declined  in  ranking  when  total  FYDP  costs 
were  considered.  For  both  proposals,  this  drop  was  caused  by  the  same  factor:  the 
substantial  staffing  requirements  required  to  maintain  the  proposed  work  over  all  5 
years  of  the  FYDP. 

R&D  on  dL  incurs  $500K  (24  staff-months)  for  the  first  year  but  a  total  of  $2.5M 
when  the  24  staff-months  were  maintained  over  the  entire  FYDP.  As  a  result,  this 
proposal  dropped  in  rank  from  1 0th  using  Y 1  cost  definitions  to  1 6th  using  the  FYDP 
definitions.  Similarly,  Lessons  Learned  required  24  staff-months  in  the  first  year  (i.e., 
$500K)  but  only  12  staff-months  for  the  remaining  years  in  the  FYDP  ($1 .5M  total). 
As  a  result,  this  proposal  dropped  in  rank  from  4th  using  Y1  cost  definitions  to  10th 
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using  the  FYDP  definitions.  Both  proposals  illustrate  how  sensitive  this  technique  is 
to  the  requirement  to  maintain  efforts  over  the  FYDP. 

2.  Proposals  Rated  Higher  Using  FYDP  vs.  Y1  Cost  Definitions.  Two  proposals 

(Predictive  Models  and  Team  Task  Analysis)  improved  in  ranking  when  FYDP  costs 
were  considered.  In  terms  of  benefit-to-cost  ratios,  Predictive  Models  improved  from 
14th  to  9th  and  Team  Task  Analysis  improved  from  12th  to  7th.  In  both  cases,  the 
reason  for  this  upward  shift  was  the  obverse  of  reason  for  the  downward  shift 
described  previously.  That  is,  both  Predictive  Models  and  Team  Task  Analysis  were 
conceived  as  1-year  projects  with  no  follow-on  costs. 


Conclusions 


The  overall  conclusion  from  this  effort  is  that  the  application  of  cost-benefit  analytic 
methods  resulted  in  sensible  alternative  plans  for  a  TRADOC  R&D  program  in  Army  training 
and  education.  The  costing  components  of  the  methodology  are  well-understood  traditional 
elements  of  cost-benefit  analyses.  The  innovative  aspect  of  the  method — the  benefits’ 
components — had  a  notable  role  in  its  apparent  success.  The  innovation  was  to  quantify  the 
benefit  by  parsing  it  into  two  subcomponents:  impact  of  the  effort  and  probability  of  its  success. 
The  multiplicative  combination  of  these  subcomponents  can  be  interpreted  as  the  “expected 
impact”  of  the  proposed  R&D  efforts. 

This  report  describes  the  method  in  some  detail  to  promote  its  application  using  different 
data.  For  instance,  TRADOC  may  wish  to  alter  the  cost  or  benefit  values  and  rerun  the  analysis 
on  the  proposed  R&D  projects.  For  the  present  application,  all  estimates  were  provided  by  the 
R&D  proposal  developers.  If  TRADOC  (or  other  potential  users)  were  to  provide  its  own 
estimates,  specific  expertise  should  be  sought  to  estimate  cost  (e.g.,  business  or  economics)  vs. 
benefit  values  (e.g.,  military  technology  or  science).  The  benefit  determination  should  also  be 
decomposed  into  the  two  subcomponents:  Military  experts  who  understand  the  effects  of 
technology  on  military  operations  should  determine  the  impact  of  a  proposed  R&D  effort,  and 
scientists  who  can  judge  the  technical  feasibility  of  proposals  should  estimate  the  probability  of 
success.  However,  actually  performing  such  judgments  and  achieving  a  consensus  of  opinion  are 
themselves  nontrivial  exercises. 

Potential  users  may  also  want  to  modify  the  objects  of  analysis  (i.e.,  the  R&D  proposals). 
While  no  technical  impediment  exists  to  such  a  reanalysis  of  the  work,  the  authors  of  this  report 
believe  that  the  current  proposals  had  merit  before  the  analytic  method  was  applied  (i.e.,  they 
were  developed  before  the  analysis — not  to  demonstrate  it).  As  such,  the  proposals  represent  a 
synthesis  of  concepts  from  the  Army  Science  of  Learning  Workshop,  whose  participants 
included  experts  in  training  and  education  and  key  stakeholders  in  institutional  and  unit  training 
in  the  Army.  The  strength  of  this  report  is  the  demonstration  of  how  to  integrate  credible  training 
proposals  with  an  analysis  methodology  to  determine  their  value  to  the  Army. 
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LEARNING  MODEL 


Working  Group  I  of  the  Army  Science  of  Learning  Workshop  examined  learning  models 
of  large-scale  organizations  outside  of  the  Army — including  other  military  Services  (Navy  and 
Coast  Guard),  universities,  and  corporations — to  extract  best  practices  and  lessons  learned  that 
could  be  applicable  for  an  Army  learning  model.  This  model  is  descriptive  because  it  explains 
basic  learning  phenomena  and  prescriptive  because  it  provides  guidance  for  instructional 
development  and  implementation. 


Selective  Implementation  of  the  Guided  Experiential  Learning  (GEL)  Model 


Concept  and  Rationale 

The  GEL  model  is  a  structured  (i.e.,  nonconstructivist)  approach  to  instruction.  It  has 
much  in  common  with  other  structured  learning  approaches,  such  as  the  Instructional  Systems 
Design  (ISD)  model  and  the  Army’s  System  Approach  to  Training  (SAT).  The  principal 
innovation  of  GEL  is  its  integration  of  modem  cognitive  learning  theory  into  instructional  design 
and  development,  particularly  the  incorporation  of  cognitive  analysis  techniques  into  the  design 
of  instruction.  Cognitive  task  analysis  provides  a  set  of  methods  for  determining  the  knowledge 
structures  and  processes  that  underlie  competent  performance.  By  providing  prescriptions  for 
imparting  those  structures  and  processes,  cognitive  task  analysis  represents  a  method  for 
developing  and  maintaining  high  levels  of  performance. 

Some  differences  of  opinion  exist  regarding  the  appropriateness  of  GEL  to  varied  types 
of  training  content.  Some  believe  that  it  applies  to  all  types  of  training  objectives,  whereas  others 
think  that  it  is  applicable  only  to  those  objectives  that  have  substantial  cognitive  requirements. 
The  purpose  of  the  proposed  project  would  be  to  apply  the  GEL  to  a  small  but  diverse  set  of 
training  objectives.  Results  would  focus  not  only  on  metrics  related  to  the  design  and 
development  of  instruction,  but  also  on  learning  outcomes  that  result  from  the  implementation  of 
the  instruction. 

Estimated  Costs  and  Benefits 

The  principal  benefit  in  implementing  the  GEL  model  is  improved  efficiency  and 
effectiveness  of  training  processes  and  outcomes — improvement  estimates  on  the  order  of  1 0% 
to  20%. 

The  implementation  of  GEL  could  be  compressed  so  that  2  people  accomplish  this  task  in 
6  staff-months  (12  staff-months  in  all).  The  primary  cost  driver  is  the  requirement  to  keep  the 
GEL  program  current  with  research  and  development  (R&D)  trends  and  to  educate  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC)  training  developers  in  its  application.  For  these 
purposes,  at  least  1  person  would  be  required  to  devote  half  his/her  time  to  this  effort  after  the 
first  year  (i.e.,  6  staff-months  per  year  for  the  remaining  4  years  of  the  FYDP). 
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Implementation  of  GEL  could  result  in  avoiding  unnecessary  costs  because  of 
inappropriate  use  of  training  design  and  development  methods;  however,  this  is  benefit  is 
difficult  to  quantify. 


Implementation  of  a 

Human  Performance  Improvement  (HPI)  Program  in  the  Army 


Concept  and  Rationale 

HPI  is  a  concept  for  diagnosing  and  solving  problems  in  human  performance.  It  assumes 
that  human  performance  problems  can  be  caused  by  a  lack  of  skills  and  knowledge  but  that  these 
problems  can  also  be  caused  by  deficiencies  in  job  rewards,  feedback,  equipment,  tools,  and  so 
forth.  Thus,  training  is  but  one  of  many  possible  interventions  (and  an  expensive  one  at  that)  for 
improving  human  performance.  HPI  prescribes  a  careful  analysis  of  the  problem  to  determine  the 
appropriate  intervention. 

Although  the  concept  of  HPI  is  perfectly  reasonable,  its  implementation  in  the  Army  is 
problematic.  It  considers  a  range  of  human  performance  problem  interventions  that  potentially 
cut  across  Army  organizations.  Training,  education,  or  doctrinal  interventions  are  clearly  under 
the  purview  of  TRADOC,  but  inventions  that  involve  equipment  modifications,  personnel 
assignment,  and  other  systems  would  necessarily  involve  non-TRADOC  organizations.  Thus, 

HPI  raises  fundamental  questions  for  the  Army,  including  the  following:  Who  should  perform 
the  initial  analysis?  What  authorities  should  he/she  possess?  Who  should  oversee  the 
implementation  of  the  intervention?  Such  questions  should  be  systematically  studied  to 
determine  the  appropriate  method  for  implementing  HPI  in  the  Army. 

Estimated  Costs  and  Benefits 

Evidence  from  other  Military  Services  and  from  private  industry  indicates  that  HPI 
implementation  can  have  a  tremendous  return  on  investment  (ROI)  in  terms  of  (1)  improvements 
in  performance  and  (2)  avoidance  of  costs  related  to  inappropriate  performance  interventions. 
However,  to  realize  such  returns,  the  Army  would  have  to  institutionalize  the  HPI  process,  which 
could  include  significant  costs.  The  proposed  study  would  identify  and  compare  alternative 
approaches  to  implementing  the  HPI  process  in  the  Army. 

Because  the  cost  and  the  ROI  for  HPI  implementation  are  so  great,  the  present  proposal  is 
limited — it  constitutes  only  a  “plan  for  a  plan.”  This  preliminary  analysis  will  identify  several 
courses  of  action.  Assuming  appropriate  data  can  be  obtained,  a  formal  analysis  of  alternatives 
(AoA)  will  then  be  performed  to  determine  a  plan  of  action  that  mitigates  risk  and  maximizes 
ROI. 


The  estimated  costs  are  based  on  a  compressed  schedule  of  development  that  involves 
four  professional  analysts  who  have  expertise  in  training,  policy,  and  program  assessment.  An 
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assumption  is  that  an  initial  operating  capability  (IOC)  can  be  reached  in  6  months.  IOC  would 
be  defined  as  a  workable  plan  for  implementing  HPI — not  implementation  of  HPI  itself. 


Blending  Distance  Learning  With  Traditional  Instruction 


Concept  and  Rationale 

Distance  learning  (dL)  is  generally  recognized  as  a  cost-effective  alternative  to  traditional 
live  methods  of  delivering  instruction.  However,  some  training  programs  [e.g.,  the  Officer 
Advanced  Course  (OAC)]  require  students  to  interact  with  instructors  and  peers.  The  Army  is 
now  considering  how  best  to  blend  dL  with  traditional  instruction.  Specifically,  the  Army  asks 
the  following  questions:  How  can  dL  be  combined  with  live  instruction  to  retain  the  social 
benefits  while  reducing  training  costs?  To  what  extent  can  virtual  interaction  be  substituted  for 
live  interaction? 

In  the  short  term,  such  questions  can  be  answered  by  reviewing  relevant  literature  on 
civilian  dL — literature  that  must  be  interpreted  within  the  constraints  of  Army  training. 
Nevertheless,  this  is  the  quickest  way  to  obtain  valid  advice  on  how  dL/live  courses  can  be 
blended.  This  literature  review  may  also  reveal  topics  that  require  gathering  new  data,  and  this 
would  represent  a  significant  effort.  However,  the  present  proposal  is  only  for  the  initial 
literature  review  because  the  scope  of  any  data  gathering  effort  cannot  be  specified  before 
examining  the  literature. 

Estimated  Costs  and  Benefits 

If  such  mixed  programs  were  implemented,  the  potential  savings  in  travel  costs  and 
personnel  disruption  would  be  considerable.  This  proposal  is  limited,  however,  because 
implementation  is  not  included  as  part  of  the  effort.  On  the  other  hand,  the  proposed  work  will 
help  to  identify  ways  to  maximize  the  benefits  of  blended  learning  while  minimizing  the  costs.  A 
compressed  time  schedule  for  reaching  IOC  in  3  months  could  be  met  by  assigning  3  staff 
members  to  the  task.  IOC  is  the  delivery  of  the  literature  review  and  appropriate 
recommendations. 


Models  To  Predict  Effectiveness  and  Efficiency  of  Training  and  Education 


Concept  and  Rationale 

The  Army  should  continue  exploring  new  learning  models,  strategies,  and  tools  to 
support  the  evolution  of  its  training  system.  In  particular,  the  Army  should  develop  models  to 
predict  the  effectiveness  and  efficiency  of  training  and  education.  Such  models  should 
incorporate  the  latest  in  learning  science  research  and  theories.  Bjork’s  (2007,  1994)  reviews  of 
cognition  and  memory  effects  show  how  theories  and  research  inform  training  practices.  His 
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analyses  of  the  research  are  especially  important  because  they  lead  to  concepts  that  are  not 
intuitively  obvious.  For  instance,  he  shows  that  manipulations  that  speed  the  rate  of  learning  may 
not  support  long-term  retention  and  transfer.  On  the  other  hand,  certain  manipulations  that  may 
retard  learning  (“desirable  difficulties”)  may  enhance  and  prolong  performance  and  promote 
creative  transfer  of  training  to  unique  situations.  Such  desirable  difficulties  include  spacing 
rather  than  massing  study  sessions,  interleaving  rather  than  blocking  practice  on  separate  topics, 
varying  how  to-be-learned  material  is  presented,  reducing  feedback,  and  using  tests  as  learning 
events.  Some  manipulations  (e.g.,  reducing  feedback)  may  enhance  retention  performance  at 
virtually  no  costs,  while  others  (e.g.,  varying  instructional  conditions)  may  enhance  retention  at 
the  cost  of  prolonging  original  learning.  Thus,  the  proposed  models  must  account  for  potential 
tradeoffs  between  learning  acquisition  and  retention  performance. 

Estimated  Costs  and  Benefits 

The  proposed  models  can  be  used  to  predict  the  effects  of  such  manipulations  on 
learning,  retention,  and  transfer  to  devise  optimal  strategies  for  timing,  sequencing,  and 
scheduling  the  training  and  testing  events.  To  the  extent  possible,  the  model  should  be  derived 
from  existing  research  findings,  although  original  empirical  research  may  be  proposed  to  fill  key 
gaps.  The  work  will  require  3  staff  members  for  1  year  (36  staff-months).  Once  the  model  has 
been  developed  and  submitted  to  TRADOC,  it  should  not  incur  additional  maintenance  costs. 


Effective  and  Efficient  Knowledge  Acquisition,  Management, 
and  Sharing  Approaches  and  Tools 


Concept  and  Rationale 

The  pervasiveness  of  dL  technologies  and  tools  has  promoted  a  new  approach  to 
acquiring,  managing,  and  sharing  knowledge.  This  knowledge  management  (KM)  approach  is 
embodied  in  the  Army’s  Battle  Command  Knowledge  System  (BCKS).  Wikipedia  (itself  a 
symbol  for  the  KM  approach)  defines  BCKS  “...as  the  change  agent  for  implementing 
knowledge  management  (KM)  capabilities  into  the  training  and  military  operations  of  the  United 
States  Army”  (Battle  Command  Knowledge  System  2007,  August).  A  recent  IDA  report  (Brooks 
et  al.,  2005)  analyzed  BCKS  capabilities  and  concluded  that  it  is  a  potentially  effective  tool  for 
enhancing  learning  and  developing  situational  understanding.  On  the  other  hand,  Brooks  et  al. 
maintained  that  BCKS  systems  architecture  is  not  well  integrated  and  that  BCKS’s  knowledge 
acquisition  processes  is  not  well  related  to  traditional  forms  of  instruction  and  training.  Research 
is  needed  to  develop  an  appropriate  KM  architecture  and  tools  for  the  Army.  This  architecture 
should  acknowledge  existing  technologies  (e.g.,  BCKS)  and  how  these  technologies  can  and 
should  be  integrated  to  promote  the  effective  sharing  of  actionable  knowledge.  With 
implementation  of  the  architecture,  the  Army  may  realize  the  full  benefits  of  KM  systems  such 
as  BCKS. 
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Estimated  Costs  and  Benefits 

The  benefit  of  this  effort  will  occur  through  improvements  in  BCKS  and  other  KM 
capabilities.  The  architecture  work  would  be  substantial,  requiring  48  staff-months  to  develop 
and  1 2  staff-months  per  year  thereafter  to  maintain.  One  of  the  not-so-obvious  maintenance  costs 
would  include  the  management  of  the  culture  change  caused  by  integrating  KM  into  TRADOC 
practices. 


Life-Long  Learning  Models  With  Intelligent  Agents  for  Individualized  Guidance 


Concept  and  Rationale 

Life-long  learning  views  dL  and  related  technologies  as  the  means  through  which 
education  and  training  can  be  provided  in  numerous  contexts  and  throughout  Soldiers’  careers. 
The  contexts  go  beyond  traditional  institutional  and  unit  training  situations  and  include  learning 
opportunities  at  home,  at  the  civilian  workplace,  and  even  during  leisure  activities  and  games.  To 
provide  guidance  in  choosing  appropriate  opportunities  and  to  assess  learning  progress,  life-long 
models  include  personnel  (e.g.,  parents,  teachers,  work  supervisors)  who  perform  an  explicit 
mentoring  function.  These  mentors  provide  advice  on  all  aspects  of  learning,  but  their  primary 
training  function  is  to  prescribe  experiences  appropriate  to  the  individual’s  stage  of  learning. 

Clearly,  one-on-one  human  mentoring  is  too  expensive  to  be  practical  on  a  large  scale. 

On  the  other  hand,  a  long  history  of  research  on  technology-based  instruction  has  demonstrated 
that  many  mentoring  functions  can  be  accomplished  through  effective  individualization  of 
training.  For  instance,  Learning  Management  Systems  (LMS)  have  been  developed  to  track  an 
individual’s  progress  and  prescribe  training  exercises  appropriate  to  this  progress.  The  challenge 
is  to  expand  computer-based  mentoring  functions  beyond  particular  training  courses  or  syllabi 
and  encompass  the  whole  spectrum  of  Army  training  and  education.  The  technology  of 
intelligent  agents  provides  the  potential  to  draw  information  from  disparate  sources  and  make 
integrated  learning  prescriptions  for  individual  soldiers.  The  present  proposal  is  to  examine 
alternative  models  for  using  intelligent  agents  to  perform  such  mentoring  functions. 

Estimated  Costs  and  Benefits 

A  system  of  intelligent  agents  for  mentoring  functions  is  viewed  as  an  enhancement  of 
existing  technology-based  individualized  instruction.  The  first  step  is  to  develop  an  intelligent 
agent  model,  or  perhaps  alternative  models,  that  would  be  compatible  with  current  distance  and 
individualized  training  standards.  The  development  work  would  be  substantial,  requiring 
48  staff-months  to  execute,  but  this  work  would  be  completed  within  1  year.  Follow-up  costs 
would  be  contingent  upon  recommendations  from  the  initial  research,  but  these  costs  are 
estimated  to  be  12  staff-months  per  year  for  the  remaining  4  years  in  the  FYDP. 
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TRAIN  SOLDIERS 


TRADOC  continues  to  be  extraordinarily  successful  in  preparing  Soldiers  for  operational 
duty.  The  following  recommendations  are  not  aimed  at  “fixing”  anything  TRADOC  does. 
Rather,  they  are  intended  as  suggestions  to  help  TRADOC  (1)  capitalize  on  recent  developments 
and  analyses  emerging  from  the  science  and  technologies  of  learning  and  (2)  provide  additional 
training  more  efficiently  and  with  fewer  resources.  In  addition  to  these  recommendations,  a  few 
research  issues  were  included  as  opportunities  for  consideration. 


Systematic  Implementation  Plan  for  Distributed  Learning  (dL) 


Concept  and  Rationale 

The  concept  of  this  proposed  task  is  to  use  dL  technology  to  accelerate  training,  reduce 
costs,  reduce  personnel  requirements,  and  improve  operational  effectiveness  without  adversely 
affecting  Soldiers  or  their  families.  This  goal  will  be  realized  through  a  series  of  recommended 
actions. 

Budget  for  Up-Front  Costs.  Although  TRADOC  can  expect  significant  cost 
savings/avoidances  by  using  dL,  initial  investments  to  develop  materials  and  implement  dL 
capabilities  are  substantial.  Savings  through  dL  can  return  these  initial  costs  quickly — as  early  as 
2  years  in  some  analyses.  However,  this  will  require  resources  not  currently  budgeted  to  develop 
and  implement  dL. 

Obtaining  up-front  funding  for  dL  is  a  major  challenge  for  TRADOC  leadership. 
TRADOC  management  should  develop  analyses  that  show  the  expected  return  on  the  initial 
investment. 

Raise  Standards  and  Expectations  for  Quality.  Most  of  the  dL  benefits  derive  from 
interactivity  and  the  ability  to  tailor  instruction  to  the  learner’s  needs.  Analyses  have  shown  that 
these  capabilities,  in  addition  to  reducing  per-hour  training  costs,  allow  learners  to  achieve 
training  objectives  more  quickly  (about  two-thirds  of  the  traditional  learning  time),  thereby 
freeing  learners  earlier  for  assignment  to  duty  stations.  Analyses  have  also  shown  that  an  early 
finish  reduces  training  costs  to  a  greater  extent  than  providing  learning  at  a  distance,  which  also 
results  in  considerable  cost  savings/avoidances.  However,  using  dL  simply  as  method  by  which 
learners  can  progress  through  material  by  simply  hitting  the  “return”  key  obviates  most  dL 
benefits.  Such  training  approaches  are  becoming  increasingly  common  and  should  be 
discouraged. 

The  state  of  knowledge  shows  how  to  achieve  the  significant  benefits  promised  by  dL. 
The  state  of  practice  should  rise  to  meet  it.  TRADOC  leadership  should  insist  on  adequate 
standards  and  raise  expectations  for  dL  quality  above  the  current  norm. 
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Establish  a  Comprehensive  Cost  Model  with  Well-Defined  Cost  Elements  for  dL 
Training.  Cost  analyses  are  much  like  research — other  analysts  can  always  find  errors  of 
omission  and  commission.  Also,  if  TRADOC  decides  to  undertake  an  initiative  to  develop  and 
use  dL,  TRADOC  management  should  establish  a  standard  dL  cost  model,  sufficiently 
harmonized  with  other  TRADOC  cost  models,  that  would  allow  explicit  cost  comparisons  to  be 
made  for  different  training  approaches. 

Establish  Routine  and  Continuous  Processes  and  Procedures  To  Ensure  the  Quality  of 
dL.  TRADOC  has  processes  and  procedures  for  assessing  and  updating  non-dL  training,  and 
TRADOC  management  should  ensure  that  these  processes  and  procedures  are  extended  to  dL 
and  sustained  by  individuals  with  dL  expertise.  Periodic  formative  evaluations  of  cost  and 
effectiveness  should  be  integrated  into  all  dL  programs.  For  TRADOC  management  (and 
perhaps  all  the  Army),  establishing  these  processes  and  procedures  will  help  ensure  quality 
control  when  contracting  for  dL  development. 

Identify  and  Manage  the  Cultural,  Administrative,  and  Budgetary  Effects  of  dL.  Most 
TRADOC  management,  administrative,  and  budgetary  procedures  are  based  on  instructor-led 
classroom  learning,  with  quotas  and  specifically  scheduled  beginning  and  ending  times.  These 
procedures  create  what  might  be  described  as  a  TRADOC — if  not  Army — training  culture.  The 
anytime,  anywhere  capabilities  and  processes  of  dL  are  anomalies  for  this  culture. 

TRADOC  leadership  will  be  required  to  incorporate  dL  into  its  training  culture. 
TRADOC  management  should  undertake  to  identify  the  specific  effects  of  dL  on  existing 
training  practices,  procedures,  and  policies  and  on  the  ability  of  TRADOC  to  incorporate  dL  as  a 
standard,  fully  accepted  training  approach.  In  this  effort,  TRADOC  management  should 

•  Establish  incentives  and  rewards  for  early  adopters 

•  Select  materials  for  conversion  to  dL  or  development  of  dL  at  the  learning-module 
level  rather  than  the  course  level 

•  Prepare  for  and  manage  second-  and  third-order  dL  effects 

•  Establish  processes,  procedures,  and  incentives  for  supervisors  to  use  in  dL  training 

•  Ensure  sufficient  coordination  between  training  and  personnel  functions  so  that 
Soldiers  who  begin  and  end  programs  of  instruction  at  different  times  receive 
assignments  quickly  to  appropriate  duty  stations 

•  Establish  training  programs  for  learners,  instructors  (“train  the  trainer”),  and  training 
developers. 

Estimated  Costs 

Taken  as  a  whole,  the  recommended  actions  would  require  about  34  staff-months  to 
develop  through  the  first  year  and  then  2  staff-months  a  year  to  maintain  thereafter.  The 
individual  recommendations  break  down  as  follows: 
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•  Creating  a  budget  for  dL  up-front  costs  will  be  relatively  minor.  The  major  costs  will 
be  to  obtain  the  funding  required  by  the  budget.  Costs  to  create  the  budget  and  obtain 
funding  are  estimated  to  all  be  about  6  staff-months. 

•  Costs  to  devise  dL  quality  standards  will  be  relatively  minor.  The  major  costs  will 
arise  from  the  effort  and  vigilance  needed  to  have  quality  standards  established  and 
maintained.  Costs  will  be  about  2  staff-months  for  the  first  year  and  1  staff-month  a 
year  thereafter. 

•  Most  of  the  cost  modeling  effort  will  involve  reaching  consensus  about  the  cost 
elements  among  the  many  stakeholders,  including  include  how  these  elements 
should  be  defined.  Costs  to  reach  the  necessary  consensus  will  be  at  least  14  staff- 
months  for  the  first  year  but  about  1  staff-month  per  year  thereafter  to  maintain  and 
modify  the  model. 

•  The  quality  control  of  courseware  should  require  about  1  staff-month  for  each  dL 
course.  For  costing  purposes,  we  assume  that  the  technique  is  tested  on  5  courses 
that  have  been  carefully  selected  to  represent  a  variety  of  content,  audiences,  and 
training  technology.  Total  costs  would  be  6  staff-months. 

•  Identifying  and  managing  effects  of  dL  should  be  a  one-time  cost  of  6  staff-months. 


Research  and  Development  on  Distributed  Learning  (dL)  Topics 


Concept  and  Rationale 

TRADOC  leadership  and  management  should  encourage  the  establishment  of  R&D 
programs  to 


•  Guide  the  development  and  selection  of  effective  dL  strategies  for  tailoring  training 
content,  sequence,  and  pace  to  the  needs,  objectives,  and  prior  knowledge  of 
individual  learners 

•  Determine  ways  to  balance  and  blend  remotely  provided  dL  and  face-to-face 
instruction 

•  Develop  and  deliver  dL  on  rugged,  handheld  devices 

•  Develop  and  provide  practicable  dL  “authoring”  (for  instructional  content)  and 
“composing”  (for  simulation  and  scenario  generation)  tools  for  use  by  local 
commanders 

•  Reduce  the  cost  of  dL  development  “authoring”  and  “composing”  through  various 
means,  including  the  development  and  use  of 

Sharable  instructional  objects 
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Capabilities  that  imbue  dL  with  sufficient  “intelligence”  to  manage  and  generate 
training  interactions  on  demand  and  in  real  time. 

Estimated  Costs 

Setting  up  an  R&D  effort  for  dL  capability  should  require  24  staff-months  for  Y1  and 
24  staff-months  per  year  for  the  remaining  4  years  in  the  FYDP. 


Analyses  of  Training 


Concept  and  Rationale 

The  intent  of  this  proposal  is  to  perform  analyses  of  training  inputs  and  outputs  to 
accelerate  training  and  reduce  training  costs  and  personnel  requirements.  The  types  of  analyses 
are  described  below. 

Establish  Roles  and  Responsibilities  and  Guidelines  To  Balance  TRADOC 's  Institutional 
Responsibility  With  the  Training  Needs  of  Operational  Forces.  TRADOC  management  should 
undertake  an  initiative  in  coordination  with  the  operational  forces  to  develop  roles  and 
responsibilities  and  establish  them  as  generally  applicable  guidelines.  A  process  to  review  these 
guidelines  periodically  and  update  them  as  needed  should  be  established  and  enforced.  At  a  more 
specific  level,  TRADOC  management  should  determine  the  extent  to  which  40/1 1  warrior  tasks 
and  drills  must  be  trained  in  Initial  Entry  Training  (IET),  assuming  the  availability  of  the  Army 
Force  Generation  (ARFORGEN)  program. 

Establish  a  Program  to  Develop,  Identify,  and  Measure  Mission-Essential  Competencies 
for  Army  Operations.  The  current  program  of  developing,  identifying,  and  measuring  essential 
tasks  is  a  necessary  and  major  contribution  to  Army  training  and  effective  operation.  However,  it 
is  reductionist.  Ways  are  needed  to  reassemble  tasks — essential  and  otherwise — into  operational 
capabilities.  The  Air  Force  has  dealt  with  this  issue  by  establishing  Mission  Essential 
Capabilities  (MECs).  The  Army  might  benefit  from  a  similar  approach  suitable  for  its 
operational  requirements. 

TRADOC  management  should  investigate  the  feasibility  and  value  of  establishing  MECs 
for  Army  personnel  and  collectives.  If  such  an  approach  proves  feasible  and  valuable,  TRADOC 
management  should  develop  and  implement  it. 

Establish  a  Program  to  Assess  the  Military  Value  of  Training.  How  much  is  a  pound  of 
training  worth?  Training  is  assumed  to  improve  the  Army’s  operational  capabilities;  however, 
the  extent  to  which  it  does  so  is  seldom  quantified.  Because  training  must  compete  for  resources 
with  other  capabilities  whose  contributions  are  more  readily  quantifiable,  it  may  be  literally  and 
frequently  shortchanged.  TRADOC  management  should  undertake  analyses  to  determine, 
quantitatively,  the  military  value  of  training  (i.e.,  to  determine  the  degree  to  which  different  sorts 
of  training  and  training  content  contribute  to  operational  effectiveness). 


A-14 


Estimated  Costs 


The  overall  cost  for  a  training  analysis  program  would  be  about  14  staff-months  to 
establish  the  program  and  8  staff-months  per  year  thereafter  to  sustain  the  program.  Costs  for  the 
individual  R&D  projects  would  be  as  follows: 

•  The  process  for  establishing  individual  roles  and  responsibilities  should  cost  about 
12  staff-months  to  set  up  and  6  staff-months  per  year  thereafter  to  sustain. 

•  The  program  to  develop  MECs  should  cost  about  1  staff-month  per  year  thereafter  to 
establish  and  maintain. 

•  Costs  for  assessing  the  military  value  of  training  are  especially  difficult  to  estimate. 
Our  best  estimate  is  that  it  should  cost  about  2  staff-months  to  establish  the  program 
and  1  staff-month  for  each  of  2  years  thereafter. 


Implementation  of  Enhanced  Training  Capabilities 


Concept  and  Rationale 

The  goal  of  this  proposal  is  to  develop  capabilities  to  accelerate  training  and  reduce 
training  costs  and  personnel  requirements.  Two  specific  capabilities  are  described  below. 

Provide  Assignment-Oriented  Training  (AOT)  or  Tailor  Existing  Training  to  an 
Individual 's  Current  or  Pending  Duty  Assignment.  TRADOC  leadership  should  increase  the 
integration  of  personnel  and  training  functions  to  establish  an  effective  AOT  program.  TRADOC 
management  should  then  coordinate  personnel  and  training  databases  so  that  an  individual’s 
training  can  be  linked  to  his/her  current  or  pending  duty  assignments.  To  reduce  course 
completion  times  and  costs,  this  tailoring  should,  to  some  appreciable  degree,  be  based  on 
assignment,  equipment,  and/or  theater  requirements. 

Establish  Guidelines  To  Provide  Resident  Institutional  Training  in  Units.  Various  means 
(e.g.,  mobile  training  teams,  correspondence  instruction,  video-tele-training,  Web-based 
instruction,  dL)  are  becoming  available  for  resident  institutional  training  in  units.  TRADOC 
management  should  establish  and  apply  guidelines  for  determining,  on  a  case-by-case  basis,  the 
most  cost-effective  approach  for  providing  training  that  meets  specific  individual  and  unit  needs. 

Estimated  Costs 

This  proposal  should  require  about  1 8  staff-months  to  establish  and  3  staff-months  per 
year  thereafter  to  maintain.  The  individual  efforts  break  down  as  follows: 
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•  To  establish  a  working  relationship  between  personnel  and  training  functions  should 
require  about  12  staff-months  for  the  necessary  procedures  and  2  staff-months  per 
year  thereafter  to  maintain  them. 

•  To  provide  resident  institutional  training  in  units  should  require  about  6  staff-months 
to  establish  the  guidelines  and  1  staff-month  per  year  thereafter  to  maintain  and 
upgrade  the  guidelines. 


Guidelines  for  Tools  and  Methods  To  Support  Training 


Concept  and  Rationale 

The  concept  of  supporting  tools  and  methods  is  to  reduce  the  “demand  side”  of  training 
through  increased  use  of  technology  integrated  with  training  design  and  objectives.  The 
technologies  suggested  are  discussed  below. 

Performance  and  Decision  Aids.  Training  requirements  can  be  reduced  to  some  extent  by 
providing  performance  and  decision  aids,  such  as  voice-interactive  maintenance  aids,  embedded 
operating  instructions,  electronic  manuals,  and  hand-held  advisors.  TRADOC  management 
should  develop  guidelines  to  identify  and  apply 

•  The  most  cost-effective  tradeoffs  between  training  objectives  and  the  provision  of 
duty-site  performance  and  decision  aids. 

•  Adjustments  that  should  be  made  in  training  objectives  and  programs,  assuming  the 
availability  of  performance/decision  aids 

•  The  proper  mix  of  on-site  personnel  who  will  rely  on  these  aids  and  those  who  will 
not. 

Tools  for  Soldier  Self-Assessment  and  Self-Directed  Learning.  Given  the  many 
capabilities  for  distributing  training  (including  but  not  limited  to  dL),  a  Soldier  could  and  should 
be  expected  to  take  more  responsibility  for  his/her  professional  growth  and  development. 
However,  he/she  needs  tools  to  do  this.  TRADOC  management  should  provide  these  tools, 
which  could  be  made  available  in  various  formats  (e.g.,  paper,  Web,  dL  capabilities)  to  all 
Soldiers.  These  tools  should  permit  Soldiers  sufficient  access  to  personal  and  training  resource 
files  to  allow  them  to  assess  their  needs  for  continuing  career  and  professional  growth  and  to 
match  these  needs  to  the  training  opportunities  available  to  them  from  TRADOC  or  any  other 
provider. 

Continuing  Education  Units  (CEUs).  Related  to  the  previous  proposal,  TRADOC 
leadership  should  encourage  professional  growth  by  requiring  all  Soldiers  to  complete  a  program 
similar  to  the  CEU  programs  offered  by  business  organizations  and  academic  institutions. 
TRADOC  management  should  establish 
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•  CEU  equivalencies  for  TRADOC  training 

•  Sufficient  coordination  between  training  and  personnel  functions  to  ensure  that 
Soldiers  receive  credit  for  and  assignments  commensurate  with  their  training 
achievements. 

Estimated  Costs 

The  various  tools  and  methods  for  supporting  training  require  about  48  staff-months  to 
develop  and  6  staff-months  per  year  thereafter  to  maintain.  The  costs  of  the  individual  methods 
break  down  as  follows: 

•  Costs  should  be  about  1 2  staff-months  to  establish  the  guidelines  related  to 
performance  and  decision  aids  and  about  1  staff-month  per  year  to  maintain  them. 

•  Costs  should  be  about  1 8  staff-months  to  establish  tools  for  Soldier  self-assessment 
and  self-directed  learning  in  the  first  year  and  1  staff-month  per  year  thereafter  to 
maintain  them. 

•  The  capability  to  provide  and  administer  CEUs  should  be  about  1 8  staff-months  to 
develop  in  the  first  year  and  about  4  staff-months  per  year  thereafter  to  maintain 
them. 


New  Programs  of  Research  in  Training 


Concept  and  Rationale 

The  concept  of  this  proposal  is  that  TRADOC  leadership  and  management  should 
encourage  the  establishment  of  R&D  programs  to  address  a  variety  of  topics.  Although  the  exact 
topics  for  research  should  not  be  preordained,  the  following  examples  show  the  types  of  topics 
that  could  be  addressed. 

Developing  a  Full  “ System-of-Systems  "  Approach.  Develop  a  systems-of-systems 
approach  for  allocating  resources  to  training.  This  capability  should  identify  optimal  allocations 
of  resources  among  such  functions  as 

•  Selection  and  training 

•  Personnel  job  classification  and  training 

•  Ergonomic  design  and  training 

•  Job  design  and  training. 

Preparing  for  the  Unexpected.  Prepare  Soldiers  to  recognize  and  successfully  handle 
unexpected  situations.  This  work  should  address  issues  of  increasing  Soldier  adaptability  but 
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should  also  attend  to  capabilities  (e.g.,  creativity,  innovativeness,  and  the  general  capability  to 
“think  outside  the  box”)  needed  for  successful  adaptations. 

Documenting  Cases.  Permit  commanders  to  record  “cases”  (stories  of  actual  military 
operations)  rapidly  and  easily  into  a  computerized  case  database,  so  that 

•  Principles  and  lessons  learned  can  automatically  be  extracted 

•  Trainers  can  rapidly  incorporate  cases  and  principles  into  programs  of  instruction 

•  Other  commanders  can  use  these  cases  when  planning  operations. 

Develop  Cognitive  Analyses.  More  fully  develop  the  processes  of  cognitive  task  analysis 
and  cognitive  readiness  assessment  for  individuals  and  teams  so  that  these  processes  reliably 
produce  the  same  results  in  all  hands. 

Improve  Performance  Measurement.  Identify  and  develop  reliable  (stable),  valid 
(relevant),  and  precise  (discriminating)  measures  of  individual  and  team  proficiencies  and 
capabilities  from  performance  in  simulations. 

Estimated  Costs 

TRADOC  should  establish  a  small  (1-2  person)  office  to  manage  these  and  later  training 
research  projects  of  immediate  interest  and  value  (about  1 8  staff-months  per  year).  The  office 
should  manage  an  annual  budget  supporting  about  $9M  of  R&D  efforts  per  year. 


Existing  and  Evolving  Strategies  for  Tailoring  Training 


Concept  and  Rationale 

The  notions  of  tailored  training  strategies,  real-time  diagnostic  evaluation,  and 
on-demand  learning  are  intimately  tied  to  dL.  Training  can  be  tailored  without  the  use  of 
technology  (e.g.,  Keller,  1968)  but  only  through  the  use  of  technology  does  it  become 
economically  feasible.  Before  the  advent  of  affordable  computer  technology,  tailored  training 
and  individualized  instruction  were  viewed  as  instructional  imperatives  but  economic 
impossibilities  (Scriven,  1975;  Keller,  1985).  With  computer  technology,  tailored  training  and 
individualized  instruction  are  now  affordable  (Fletcher,  1992;  1997;  2006).  dL  is  an  effort  to 
capitalize  on  this  affordability. 

Even  without  tailored  training,  some  benefits  of  dL  remain  (e.g.,  the  ability  to  deliver 
training  and  performance  aiding  anytime,  anywhere).  However,  data  show  that  the  benefits  of  dL 
are  minor  compared  to  the  training  time  and  resource  savings  to  be  gained  from  training  tailored 
to  the  needs,  abilities,  duty  assignments  and,  especially,  prior  knowledge  of  learners  (Foster  and 
Fletcher,  2003;  Dodds  and  Fletcher,  2004;  Wisher  and  Fletcher,  2004;  Fletcher,  2006).  Savings 
and  cost  avoidances  from  dL  with  tailored  training  can  reduce  training  time  alone  by  30%  to 
50%.  Such  training  would  allow  TRADOC  to  continue  to  fulfill  its  mission  despite  downward 
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budget  pressures.  Beyond  savings  and  costs,  of  course,  the  real  payoff  of  tailored  training  is 
enhanced  operational  readiness  and  effectiveness  through  ensured  human  performance  whenever 
and  wherever  it  is  needed. 

Development  of  the  techniques  to  use  computer  technology  to  tailor  instruction  began  in 
the  1950s.  By  the  early  1970s,  the  state  of  the  art  provided  reliable  techniques  to 

•  Adjust  a  learner’s  rate  of  progress,  allowing  as  much  or  as  little  time  needed  to  reach 
specified  objectives 

•  Tailor  the  content  and  the  sequence  of  instructional  content  to  each  learner’s  or 
group  of  learners’  needs 

•  Adjust  the  level  of  instruction,  making  it  as  easy  or  difficult,  specific  or  abstract, 
applied  or  theoretical,  as  necessary 

•  Adjust  to  learners’  most  efficient  learning  styles  (collaborative  or  individual,  verbal 
or  visual,  and  so  forth). 

These  capabilities  have  been  described,  discussed,  and  reviewed  by  Galanter  (1959), 
Atkinson  and  Wilson  (1969),  Suppes  and  Momingstar  (1972),  Fletcher  and  Rockway  (1986),  and 
many  others.  Unfortunately,  the  state  of  practice  in  this  area  has  lagged  far  behind  the  state  of  the 
art.  TRADOC  should  begin  an  effort  to  apply  these  proven  techniques  systematically. 

Estimated  Costs 

Although  more  research  is  needed  and  will  prove  to  be  of  value,  the  techniques  to  tailor 
training  now  exist.  The  expense  is  in  applying  them,  and  the  scope  of  application  will  determine 
the  cost.  However,  R&D  cited  in  the  literature  suggests  that  cost  savings  and  avoidances  realized 
by  using  these  strategies  will  significantly  exceed  the  costs  of  implementation. 

The  approach  suggested  is  to  apply  all  these  methods  to  a  single  program  of  instruction  (POI) 
that  experienced  trainers  deem  appropriate  for  tailored  training.  Initial  start-up  costs,  which 
involve  developing  and  implementing  the  POI,  are  estimated  to  be  low  (12  staff-months)  because 
the  methods  are  known.  Follow-on  work  (3  staff-months  per  year  thereafter)  to  maintain  the 
program  and  to  assess  its  effectiveness  is  required. 


Existing  and  Evolving  Strategies  for  Real-Time  Diagnostic  Evaluation 


Concept  and  Rationale 

Evaluations  of  learners  and  learner  progress  are  commonly  and  intermittently  performed 
using  explicit  tests,  instructor  assessments  or,  even,  after  action  reviews  (AARs).  These 
evaluations  are  worthwhile,  if  not  essential.  However,  management  of  any  process,  including 
learning  processes,  benefits  from  currency  and  the  ready  availability  of  progress  assessment.  The 
use  of  technology  in  training  makes  continuous,  unobtrusive  assessment  of  learner  progress 
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possible,  and  the  use  of  tailored  training  techniques,  especially  those  referenced  previously, 
require  such  assessment  (Fletcher,  2002).  Just  as  with  the  one-on-one  instruction  provided  by 
human  tutors,  technology  used  for  diagnostic  evaluation  is  based  on  models  of  the  learner.  The 
models  can  be  qualitative  or  quantitative,  explicit  or  implicit,  pre-existing  or  constructed  on  the 
fly,  but  they  must  be  available  in  some  form  to  enable  technology-based  evaluations  (Bruner, 
2004;  Fletcher,  1975;  Foster  and  Fletcher,  2003). 

As  with  strategies  for  tailoring  instruction,  the  state  of  practice  in  this  area  has  lagged  far 
behind  the  state  of  the  art.  Capabilities  being  discovered  and  rediscovered  in  areas  now  covered 
by  Intelligent  Tutoring  Systems  (ITSs)  seem  particularly  promising.  TRADOC  should  begin  a 
systematic  effort  to  apply  proven  techniques  for  real-time,  diagnostic  assessment.  In  addition, 
more  and  better  ways  to  use  the  timing,  display,  and  computational  capabilities  of  computers  for 
more  precise  diagnostic  models  of  learners  seem  ripe  for  development.  TRADOC  should 
encourage  R&D  to  exploit  and  apply  such  capabilities  fully  through  explicit  testing  or  human- 
computer  interactions. 

Estimated  Costs 

Although  more  research  about  diagnostic  evaluation  is  needed  and  will  prove  to  be  of 
value,  these  techniques  now  exist.  The  cost  is  in  applying  them,  and  the  scope  of  application  will 
determine  the  amount.  Initial  costs  are  estimated  at  36  staff-months  for  the  first  year  and  12  staff- 
months  per  year  for  the  remaining  4  years  in  the  FYDP.  In  addition,  the  R&D  cited  in  the 
literature  suggest  that  cost  savings  and  avoidances  realized  by  using  these  strategies  will 
significantly  exceed  the  costs  to  implement  them. 


Rapid  Conversion  of  Lessons  Learned 


Concept  and  Rationale 

For  today’s  rapidly  evolving  operational  environments,  lessons  learned  in  theater  are  of 
critical  importance  in  training.  Technology  can  and  should  be  exploited  to  improve  the 
adaptability  and  agility  of  individuals,  teams,  and  units.  Capabilities  that  bring  us  increasingly 
closer  to  this  goal  are  being  developed.  Among  these  capabilities  are  those  developed  by  the 
Representing  Enriched  CONtext  (RECON)  project.  Such  projects  enable  personnel  in  theater  to 
enter  their  experiences,  or  “war  stories,”  quickly  and  easily  into  databases  that  can  be 
“understood”  and  searched  on  a  computer.  Once  these  experiences  have  been  entered,  the 
computers,  using  techniques  developed  by  machine  learning  research  efforts,  can  abstract 
principles  from  them  for  ready  application  to  training,  performance  aiding,  and  decision  aiding. 

TRADOC  should  begin  a  systematic  effort  to  apply  these  capabilities  in  training  as  a  way 
to  maximize  institutional  learning  adaptability  and  improve  linkages  between  the  generating 
force  and  the  operating  force. 
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Estimated  Costs 


The  proposed  project  will  examine  the  various  channels  through  which  lessons  learned 
are  gathered  and  processed  and  the  types  of  information  obtained  from  them.  This  information 
will  be  used  to  develop  data-packaging  methods  (e.g.,  meta  tags)  that  are  compatible  with  these 
various  channels  and  provide  information  on  which  training  developers  can  act. 

The  Y1  start-up  costs  (24  staff-months)  cover  the  effort  to  develop  these  methods, 
whereas  the  follow-on  funding  (12  staff-months  per  year  for  the  remaining  4  years  in  the  FYDP) 
is  for  implementing  the  methods  in  an  operational  environment. 
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DEVELOP  LEADERS 


For  several  years,  the  U.S.  Army  has  recognized  the  need  to  develop  self-aware,  adaptive 
leaders  who  can  perform  effectively  in  a  broad  range  of  situations  across  the  full  spectrum  of 
operations.  This  working  group  examined  what  we  know  about  adaptability  and  explored 
possible  strategies  for  accelerating  the  growth  of  key  leader  skills  that  are  thought  to  support 
adaptability.  The  group  also  discussed  some  of  the  organizational  factors  (socialization, 
cohesion)  that  help  to  establish  the  climate  within  which  adaptable  leaders  perform. 


Leader  Socialization 


Concept  and  Rationale 

A  recent  study  about  the  development  of  Army  leaders  says  that  leaders  should  “...create 
a  command  climate  that  supports  operational  excellence  and  also  motivates  competent  people  to 
continue  their  military  service”  (Ulmer,  Shaler,  Bullis,  DiClemente,  and  Jacobs,  2004).  To  create 
that  climate,  this  study  highlights  two  leader  behaviors  involving  socialization  skills  of  major 
significance: 

1 .  Gets  out  of  the  headquarters  and  visits  the  troops 

2.  Builds  and  supports  teamwork  within  staff  and  among  units. 

Officers  who  participated  and  responded  included  the  Commanders,  Assistant  Division 
Commanders,  Chiefs  of  Staff,  8  members  of  each  Division  Staff,  and  from  6  to  10  subordinate 
commanders  in  each  Division.  They  rated  the  socialization  items  above  as  “most  important  for 
setting  climate”  and  “behaviors  for  Division  Commanders  to  work  on.”  The  question  here  is  how 
such  socialization  skills  can  be  enhanced  in  officer  career  development. 

Leader  socialization  with  troops  is  different  from  Soldier  socialization  in  peer  groups. 
Peer  groups  have  day-to-day  activities  and  participation  in  classes  and  in  units  for  getting 
acquainted  and  developing  cohesion.  In  contrast,  leaders  need  to  make  themselves  available  to 
their  troops  to  foster  respectful  socialization,  and  that  challenge  is  increasingly  difficult  because 
of  the  emphasis  on  distributed  operations,  electronic  networks,  and  larger  (remote)  areas  of 
responsibility.  One  way  to  proceed  is  to  take  advantage  of  electronic  communication  as  a  means 
of  helping  instead  of  hindering  the  socialization  process. 

Research  with  peer  groups  (Blanch,  Orvis,  and  Wisher,  2003)  shows  that  Web-based 
communication  before  face-to-face  meetings  can  accelerate  the  socialization  process.  The 
technique  was  demonstrated  in  a  trial  application  for  reserve  component  officers  enrolled  in  the 
Armor  OAC.  It  worked  by  using  text  messaging  during  a  block  of  instruction  before  resident 
coursework.  A  similar  technique  should  be  explored  to  determine  the  benefits  provided  to 
Commanders  who  routinely  exchange  information  with  their  Soldiers.  These  should  be  day-to- 
day  communications  and  not  only  “special  messages  from  your  Commander.”  Topics  (e.g., 
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updates  on  daily  activities  and  staff  meetings)  should  encourage  similar  information  exchanges 
from  the  Soldiers.  These  exchanges  would  provide  a  foundation  for  more  meaningful  face-to- 
face  visits.  Collaborative  technologies  (e.g.,  text  messaging)  for  influencing  socialization  warrant 
further  investigation,  particularly  in  combination  with  the  ARFORGEN  model  that  includes  a 
form  of  force  stabilization.  Force  stabilization  emphasizes  Leader  socialization  with  the  Soldiers 
as  an  important  way  ensure  a  good  long-term  working  relationship. 

Estimated  Costs 

The  cost  of  this  study  is  estimated  at  about  1 8  staff-months  plus  costs  of  implementation 
per  course  or  application,  estimated  at  8  staff-months.  For  costing  purposes,  a  single  test  course 
is  assumed  to  be  developed  for  testing  purposes.  The  study  involves  two  phases:  (1)  assess  the 
benefits  of  collaborative  technologies  for  classroom  or  Commander-unit  socialization  and 
(2)  verify  its  utility  through  interviews  with  officers  who  had  the  socialization  experience  and 
those  who  did  not.  Implementation  would  incur  the  cost  of  configuring  networks  for  day-to-day 
within-group  communications,  designing  computer-based  introductions  to  new  assignments  or 
courses,  and  combining  these  with  collaborative  technologies. 

Costs  pertain  to  coordinators  required  to  configure  a  specific  socialization  capability  and 
to  educate  users.  This  cost  would  be  applicable  for  each  implementation.  If  successful,  the  Army 
could,  with  little  to  no  added  expense,  introduce  the  approach  broadly  using  available  computer 
equipment. 


Critical  Thinking  Skills 


Concept  and  Rationale 

Ample  evidence  exists  that  critical  thinking  skills  (CTS)  have  value  for  battle  command 
and  tactical  decision  making,  particularly  when  uncertainty  and  time  pressure  are  factors.  At  the 
same  time,  according  to  an  Army  workshop  report,  Training  Critical  Thinking  Skills  for  Battle 
Command  (Reidel,  Morath,  and  McGonigle,  2001),  little  consensus  exists  on  what  exactly 
constitutes  critical  thinking  and  how  it  should  be  measured.  This  proposal  suggests  that  the  Army 
intensify  its  efforts  to  implement  training  of  critical  thinking  based  on  experience  gained 
previously. 

In  general,  we  can  characterize  critical  thinking  as  an  ability  to  assess  one’s  own  thinking 
and  the  thinking  of  others  to  gain  understanding  and  achieve  wise(r)  judgments.  Ample 
descriptions  about  critical  thinking’s  concepts,  values,  and  principles  to  help  inform  our  learning 
strategies  supplement  this  broad  definition.  More  difficult,  however,  is  the  development  of  good 
methods  and  procedures  for  learning  and  performing  critical  thinking. 

A  plausible  strategy  for  developing  CTS  combines  information-based  instruction  on  the  concepts 
(knowledge),  demonstration  of  the  processes  (performance-based),  and  guided  practice 
(coaching/feedback)  with  realistic  problems.  Yet,  the  primary  implementation  information  (e.g., 
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Cannon-Bowers  and  Salas,  1998;  Gerras,  2006;  and  Reidel  et  al.,  2001)  is  mostly  concerned  with 
theories  and  general  procedures  for  how  to  develop  critical  thinking.  Far  fewer  specifics  are 
available  about  what  to  do  and  what  works. 

An  Army  example  of  good  instructional  principles  is  the  Think  Like  A  Commander 
(TLAC)  vignette  program  to  train  adaptive  leaders  from  the  Brigade  Commander  down.  It 
presents  a  simulation  of  a  situation  to  a  commander  in  the  context  of  a  general  problem  that  he 
has  experienced  previously  but  with  a  different  specific-problem  situation.  The  commander 
works  through  his  interpretation,  his  reasoning,  and  the  types  of  possible  actions.  Information  is 
provided  to  relate  that  thinking  back  to  the  problem-solving  thinking  processes  of  great 
commanders.  Through  a  series  of  vignettes,  a  commander  starts  to  think  more  about  the  process 
of  resolving  problems  while  he  practices  critical  thinking. 

Using  the  same  kind  of  teaching  strategy  represented  by  TLAC,  the  Army  has  begun 
adding  more  capability  to  foster  critical  thinking.  An  initial  implementation  has  begun  at  the  U.S. 
Military  Academy,  West  Point.  In  addition,  several  Army  schools — Army  Management  Staff 
College  (AMSC),  Command  and  General  Staff  College  (CGSC),  Army  War  College  (AWC)— 
have  explored  various  curricula  for  critical  thinking.  The  Center  for  Army  Leadership  (CAL)  has 
a  current  effort  developing  measures.  A  recent  effort  just  completed  by  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  uses  Web-based  training  to  focus  on  eight 
CTS.  Each  skill  module  is  2  hours  long  although  longer  versions  (8  to  10  hours)  for  four  of  the 
skills  are  available.  This  product  is  ready  to  deliver  now. 

Teaching  critical  thinking  is  expanding  in  the  Army.  However,  four  recommendations 
(about  what  to  teach)  presented  originally  in  the  Workshop  about  training  critical  thinking 
(Reidel  et  al.,  2001)  need  added  attention: 

1 .  Train  interpersonal  skills  for  the  application  of  critical  thinking.  One  of  the  barriers 
to  the  use  of  critical  thinking  is  social  awkwardness.  Learning  about  critical  thinking 
is  important,  but  its  use  will  be  limited  unless  others  accept  it  and  one’s  command 
supports  it. 

2.  Model  the  use  of  critical  thinking  skills.  Training  should  immerse  students  in  the 
critical  thinking  process,  either  through  working  with  someone  who  is  critically 
thinking  or  by  showing  them  how  to  work  through  the  problem  using  a  set  of 
principles.  Thus,  facilitators  must  be  good  critical  thinkers  and  receive  training  and 
feedback  in  critical  thinking. 

3.  Train  for  transfer  of  training.  Even  using  simulations,  all  possible  situations  cannot 
be  simulated.  Methods  for  facilitating  transfer  of  training  to  other  situations  are 
available  and  should  be  used  when  developing  critical  thinking  training  programs. 

4.  Include  critical  thinking  training  in  every  course  taught  in  the  Army.  This  can  be 
accomplished  easily  on  one  level  by  emphasizing  elements  (e.g.,  asking  open-ended 
questions  and  reinforcing  the  concept  of  working  through  problems)  that  promote 
critical  thinking.  Far  more  difficult  is  training  someone  to  deal  with  novel  situations 
through  the  use  CTS  and  interactive  exercises  as  part  of  the  course  design. 
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Overall,  the  Army  seems  to  be  acquiring  good  experience  in  teaching  critical  thinking 
and  should  increase  efforts  to  offer  more  courses.  For  maximum  effect,  the  Army  should  blend 
its  past  experience  with  these  four  recommendations  to  improve  the  teaching  and  the  use  of 
critical  thinking. 

Estimated  Costs 

To  teach  critical  thinking  systematically,  the  cost  of  this  project  would  be  on  the  order  of 
24  staff-months  for  each  new  course  preparation.  Augmenting  a  course  with  critical  thinking 
exercises  should  be  possible  for  one-quarter  to  one-half  that  amount.  The  costing  projections 
assume  that  a  minimum  of  two  new  courses  will  be  developed  and  that  two  courses  will  be 
augmented.  Maintaining  each  course  would  require  about  1  staff-month  per  year  thereafter.  The 
benefit  would  be  improved  efficiency  and  effectiveness  not  just  for  training  of  CTS,  but  also  for 
solving  problems  on  the  asymmetric  battlefield  and  in  stability  operations. 


Experience-Based  Learning  Methods 


Concept  and  Rationale 

The  study  of  past  conflicts  and  their  successes  and  problems  is  an  integral  part  of  learning 
leader  skills.  Leaders  read  and  discuss  accounts  of  battle  strategies  and  tactics,  illustrating  points 
with  the  help  of  situation  maps,  AARs,  photos,  and  video.  Personal  accounts  from  veterans 
sometimes  add  color  and  detail.  A  newer  development  is  the  use  of  multimedia  technology  to 
enhance  the  experience,  illustrated  with  examples  from  the  Mazar-e  Sharif  campaign  in 
Afghanistan  in  2001  (Knarr  and  Richbourg,  2007). 

The  concept  is  to  develop  digital  versatile  discs  (DVDs)  with  data,  analyses,  and  lessons 
learned.  However,  the  DVD’s  built-in  capability  allows  much  more  than  that.  The  DVD  is  set  up 
to  be  conducive  to  discovery  or  directed  learning  at  multiple  levels  and  from  different 
perspectives  and  analytical  approaches.  For  Mazar-e  Sharif,  the  material  is  said  to  be  robust 
enough  to  support  nine  topics,  as  follows: 

1 .  Traditional  (as  threads  develop  during  research) 

2.  Levels  of  war  (tactical,  operational,  strategic) 

3.  Principles  of  war  (Mass,  Objective,  Offense,  Surprise,  Economy  of  Force, 

Maneuver,  Unity  of  Command,  Simplicity) 

4.  Doctrine,  Organization,  Training,  Material,  Leadership,  Personnel,  Facilities 
(DOTMLPF) 

5.  Diplomatic,  Information,  Military,  Economic  (DIME) 
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6.  Mission,  Enemy,  Terrain  and  weather,  Troops  -  Time,  Civil  (METT-TC) 
considerations  to  include  cultural  aspects 

7.  Hypothesis  testing 

8.  Case  study 

9.  Staff  rides. 

For  example,  the  second  topic — levels  of  war — can  be  used  to  analyze  various  campaign 
aspects:  political,  cultural,  alliances,  information  operations,  interdependencies,  rules  of 
engagement  (ROE),  logistics,  and  adaptability.  The  approach  allows  drilling  into  and  developing 
a  subtopic  such  as  ROE,  which  specify  the  circumstances  and  limitations  under  which  forces 
conduct  operations  other  than  war  (OOTW)  or  begin  or  continue  combat.  The  index  for  the 
Mazar-e  Sharif  example  points  to  several  video  clips,  audio  clips,  and  transcripts  that  specifically 
address  or  relate  to  ROE.  One  discussion  is  about  the  problems  associated  with  positively 
identifying  targets  from  aircraft  at  20,000  feet.  The  lead-in  to  that  is  “Boots  on  the  Ground,” 
where  the  Secretary  of  Defense  explains  the  need  for  ground  troops:  “In  Afghanistan,  precision- 
guided  bombs  from  the  sky  did  not  achieve  their  effectiveness  until  we  had  boots  and  eyes  on  the 
ground  to  tell  the  bombers  exactly  where  to  aim.”  Another  discussion  by  General  Franks  is  about 
“How  well  ROE  supported  the  targeting  process”  and  “How  he  would  consider  changing  the 
ROE  development  process  to  support  the  mission.”  Such  information,  combined  with  battle 
re-creations  using  computer-generated  virtual  worlds,  provides  a  powerful  educational  tool  for 
senior-level  conferences  and  leader  development  courses. 

The  DVD  capability  to  learn  from  experience  is  a  technology-rich  way  to  develop  leader 
skills  based  on  past  conflicts.  Multimedia  battle  re-creations,  expert  commentaries,  and 
supporting  materials  allow  leaders  to  study  and  analyze  near-real  combat  conditions  from 
command,  Soldier,  and  enemy  perspectives.  It  is  the  next  best  thing  to  being  there.  The  Army 
should  take  advantage  of  not  only  the  materials  on  Mazar-e-Sharif,  which  is  ready  for  use,  but 
should  also  develop  further  two  other  such  re-creations:  the  Battle  of  73  Easting  from  the  1991 
Gulf  War  and  the  Battle  for  Fallujah,  Iraq,  in  2004. 

Estimated  Costs 

This  implementation  is  possible  for  the  cost — estimated  at  24  staff-months — of 
developing  teaching  points  emphasizing  specific  themes  (e.g.,  cultural  sensitivity;  adaptability) 
to  accompany  a  multimedia  battle  re-creation  and  about  3  to  6  staff-months  to  IOC.  Maintenance 
costs  would  be  about  1  staff-month  per  year  thereafter.  Teaching  development  should  focus  on 
the  needs  of  particular  schools  and  be  done  in  cooperation  with  their  subject  matter  experts 
(SMEs). 
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FUTURE  CAPABILITIES 


Members  in  this  working  group  envisioned  what  learning  science  and  emerging 
technologies  will  offer  within  the  next  decade  and  articulated  how  the  Army  might  train  Soldiers 
and  develop  leaders  more  effectively. 


Improvisation 


Concept  and  Rationale 

During  the  conference,  much  discussion  centered  on  developing  “adaptive”  leaders, 
without  much  being  offered  to  specify  what  that  means.  At  the  same  time,  one  participant 
(Wayne  Hodgins)  brought  up  the  (undeveloped)  notion  of  “strategic  forgetting.”  What,  he 
wondered,  might  it  mean  if  unlearning  past  norms  of  behavior  became  useful  in  the  future ?  This 
question  rekindled  the  story  of  an  accomplished  pianist  who  was  having  a  terrible  time  learning  a 
new,  radical  piece  of  music  because  it  required  that  she  get  off  the  piano  stool,  stand  up,  walk 
around  to  the  side  of  the  piano  and  strike  the  strings  directly  with  her  fists.  Maybe  unlearning 
“norms”  is  worth  investigating. 

Most  of  the  literature  on  improvisation  seems  to  focus  on  musical  or  theatrical 
performance,  both  of  which  require  massive  content  domain  knowledge  before  the  practice  of 
improvisation  can  begin.  This  did  not  seem  helpful,  as  we  are  advised  that  adaptability  skills  in 
the  Army  have  to  be  pushed  down  the  ranks  to  less-experienced  personnel.  Researching 
“instruction  to  develop  adaptive  leaders”  yields  lots  of  information  in  Master  of  Business 
Administration  (MBA)  programs  and  in  educational  (i.e.,  school)  leadership  programs,  but, 
again,  these  appear  to  be  focused  on  extending  traditionally  fixed  bounds  of  knowledge  along 
historically  established  domains. 

Perhaps  more  interesting  is  research  done  at  the  edges  of  experience,  such  as  that 
conducted  with  individuals  who  have  autism  (Marckel,  Neef,  and  Ferreri,  2006): 

Language  and  communication  are  major  areas  of  concern  for  children  with  autism 
{Diagnostic  and  Statistical  Manual  of  Mental  Disorders,  American  Psychiatric 
Association,  2000).  Research  has  shown  that  augmentative  and  alternative 
communication  (AAC)  systems,  such  as  sign  language,  electronic  communication 
aids,  and  the  picture  exchange  communication  system  (PECS),  can  increase  the 
communicative  interactions  of  children  with  autism  and  enable  them  to  exercise 
control  over  their  environments  (e.g.,  by  making  requests). . . 

Fields  at  less-certain,  farther  edges  of  practice  such  as  this  [e.g.,  stranded  mountain 
climbers,  emergency  room  physicians,  and  National  Aeronautics  and  Space  Administration 
(NASA)  astronauts]  may  offer  more  promising  leads.  The  Wharton  School  at  the  University  of 
Pennsylvania  has  conducted  some  research  in  this  area,  defining  a  concept  of  “resiliency”  as 
being  particularly  pertinent. 
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Directly  teaching  the  art  of  “improvisation”  might  be  one  way  of  addressing  the  needed 
“adaptive”  capabilities  in  a  purposeful  manner,  rather  than  in  an  indirect  manner. 

Estimated  Costs 

Several  cohorts  would  have  to  participate  in  a  candidate  training  program  that  included 
individual  aptitude,  capability,  and  personality  tests.  The  estimated  level  of  effort  for  this  work 
would  be  about  9  staff-months.  Follow-on  field  studies  would  also  be  needed.  A  curriculum 
might  be  developed  for  as  little  as  12  staff-months.  Testing  it  would  take  a  longer  time,  but  no 
real  benefit  could  be  realized  without  such  assessments.  Testing  time  needed  would  be  at  least 
36  staff-months.  The  benefits  are  hard  to  project,  but  this  seems  to  be  something  everyone 
wants — without  much  science  to  support  the  oft-stated  desires  or  to  point  to  productive  new 
directions. 


Folksonomies 


Concept  and  Rationale 

Troops  that  benefit  from  the  “lessons  learned”  of  their  predecessors  are  far  more  likely  to 
survive  than  those  who  do  not.  Technologies  that  can  support  the  capture,  representation,  and 
accessibility  of  “lessons  learned”  will  be  of  great  use  to  newly  deployed  Soldiers.  Traditional 
knowledge-capture  and  knowledge-representation  technologies  have  emphasized  brittle 
technologies  that  can  be  exploited  only  by  research  scientists.  These  technologies  are  of  no  use  to 
individuals  who  do  not  have  highly  specialized  educations  (i.e.,  enlisted  personnel).  New  forms 
of  knowledge-capture  technologies  offer  greater  promise  in  the  form  of  “folksonomies.” 
Reporting  in  Wired.com,  Terdiman  (2005)  described  these  technologies  as  follows: 

These  days,  a  growing  number  of  sites  whose  content  is  user-created  rely  on 
tagging  systems,  also  known  as  folksonomies,  for  the  added  value  Butterfield  is 
talking  about.  Flickr  and  the  social-bookmarking  site  Delicious,  along  with  Furl, 
are  generally  considered  folksonomy  trailblazers,  but  now  sites  like  MetaFilter 
and  the  blog  index  Technorati  have  jumped  on  board,  and  more  are  expected  to 
follow. 

“It’s  very  much  people  tagging  information  so  that  they  can  come  back  to  it 
themselves  or  so  that  others  with  the  same  vocabulary  can  find  it,”  said  Thomas 
Vander  Wal,  the  information  architect  credited  with  coining  the  term 
‘folksonomy.’ 

“To  me,  they’re  a  great  new  organization  tool  for  applications  and  large  content 
sites,”  said  Matt  Haughey,  the  founder  of  MetaFilter.  “Tags  are  great  because  you 
throw  caution  to  the  wind,  forget  about  whittling  down  everything  into  a  distinct 
set  of  categories  and  instead  let  folks  loose  categorizing  their  own  stuff  on  their 
own  terms.  It  hasn’t  always  been  easy  to  get  users  to  take  on  such  responsibility. 
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But  as  more  people  understand  what  tags  are,  how  they  work,  and  why  they’re 
important,  the  number  of  participants  in  folksonomies  has  grown.” 

“Tags  are  great,  but  the  thing  that  is  hard  is  getting  people  to  use  them,”  said 
Caterina  Fake,  who  co-founded  Flickr  with  husband  Stewart  Butterfield.  “But  the 
thing  that  has  happened  recently  is  they’ve  become  part  of  a  social  arena  in  which 
they  are  valuable  not  only  to  the  individual  but  to  the  group.” 

The  Armed  Services  excel  at  developing  identity  communities.  One  could  argue, 
however,  that  these  are  communities  established  by  an  assault  on  individuality  and  that  this  has 
negative  consequences  for  future  (creative)  behavior.  Much  information  is  available  to 
recommend  the  establishment  of  team  vs.  individual  self-interest  but  as  much  information  may 
be  available  to  recommend  the  establishment  of  a  sense  of  team  created  by  professional 
capacities  that  are  different  from  those  officially  recognized  [i.e.,  in  ways  not  accommodated  by 
traditional  military  occupation  specialties  (MOSs)  and  other  military  distinctions]  and  in  ways 
that  might  be  not  possible  to  predict. 

Tools  that  would  allow  individuals  and  teams  of  deployed  personnel  to  document  and 
organize  their  experiences,  replete  with  anomalies,  personal  reflection,  speculation,  and  advisory 
commentary  in  their  own  language,  would  be  valuable  to  newly  deployed  personnel  trying  to 
keep  pace  with  ever-changing  conditions  on  the  ground.  Newly  deployed  troops  have  little 
difficulty  learning  the  “lingo”  of  the  field,  but  they  may  have  great  difficulty  learning  the 
meaning  and  nuance  of  the  language. 

Ideally,  data  captured  from  the  field  would  be  fed  back  to  residential  training  schools, 
where  students  would  engage  the  content  in  high-fidelity  training  “games.”  The  high-end 
capacity  of  computer  servers  at  the  residential  training  schools  would  provide  for  realistic 
scenarios,  imparting  a  high  degree  of  engagement  through  the  luddologic  aspect  of  game-play 
behavior  and  the  consequent,  narratologic  sense  of  self-agency  that  is  realized  in  virtual 
environments  with  highly  probable  consequences.3  This  game  environment  would  teach  the 
skills  required  by  on-line  community  development,  based  on  situational  needs.  Fielded,  hand¬ 
held  tools  could  maintain  the  task  development  activities  of  data  capture  and  taxonomy 
development  for  real-time  use  and  would  recall  benefits  of  the  narratologic  aspect  of  the 
residential  game-play  environment. 

Estimated  Costs 

An  anthropological  study  of  practices  in  use  in  the  field  (to  be  institutionalized  in 
technology)  could  be  accomplished  for  36  staff-months.  Game  development  costs  could  range 


Luddologic  and  narratologic:  Narratology  is  the  study  of  narrative  and  narrative  structure  and  the  ways  they 
affect  our  perception.  The  narrativists’  approach  video  games  in  the  context  of  what  Janet  Murray  (Professor  at 
Georgia  Tech)  calls  “Cyberdrama.”  The  narrativists’  major  concern  is  with  video  games  as  a  storytelling  medium, 
one  that  arises  out  of  interactive  fiction.  The  ludologists  break  sharply  from  this  view.  Their  perspective  is  that  a 
video  game  is  first  and  foremost  a  game  and  that  it  needs  to  be  understood  in  terms  of  its  rules,  interface,  and  in 
terms  of  the  concept  of  play.  The  term  ludology  arose  within  the  context  of  non-electronic  games  and  board  games 
in  particular  but  gained  popularity  after  it  was  featured  in  a  1 999  article  by  Gonzalo  Frasca  (a  game  designer  and 
academic  researcher).  The  name,  however,  has  not  yet  caught  on  fully.  Major  issues  in  the  field  concern  questions  of 
narrative  and  of  simulation  and  whether  video  games  are  either,  neither,  or  both. 


A-31 


from  $3M  to  $9M.  Development  of  software  for  commercial  off-the-shelf  (COTS)  hand-held 
devices  for  knowledge  capture  and  representation  could  cost  $1M  to  $2M. 


Team  Task  Analysis/Behavior  Modeling 


Concept  and  Rationale 

Much  has  been  done  for  team  or  group  task  analysis.  The  Department  of  Defense  (DoD) 
has  paid  for  most  of  it.  ARI  has  conducted  much  research  in  this  area,  but  most  of  what  has 
resulted  appears  to  be  brittle  and  of  little  help  in  being  predictive. 

Research  that  looks  to  be  effective  in  modeling  crowd  behavior  under  various 
circumstances  is  ongoing.  The  reverse  of  the  statistical  Simpson  effect4  may  be  at  work. 

This  is  an  area  in  which  many  people  are  interested  but  little  is  being  accomplished. 
Again,  closed-ended  computational  models,  the  workings  of  which  no  one  can  explain  (e.g., 
Latent  Semantic  Analysis),  may  not  have  anything  to  do  with  reality. 

Why  not  start  with  an  anthropological  model  based  on  field  studies  of  what  people 
actually  do,  how  they  describe  effectiveness,  and  so  forth? 

Estimated  Costs 

A  $1-M  effort  to  study  the  differences  between  the  use  of  anthropological  vs.  computer- 
science  models  would  be  fascinating. 


4  Simpson’s  paradox  (or  the  Yule-Simpson  effect)  is  a  statistical  paradox  wherein  the  successes  of  groups 

seem  reversed  when  the  groups  are  combined.  This  result  is  often  encountered  in  social  and  medical  science 
statistics  and  occurs  when  a  weighting  variable,  which  is  irrelevant  to  the  individual  group  assessment,  must  be  used 
in  the  combined  assessment. 
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CUMULATIVE  COSTS  AND  BENEFITS  TABLES 

Table  B-l 


Cumulative  Y1  Costs  and  Benefits 


Short  Titles 

Cumulative 
Y1  Costs 
(K  Dollars) 

Cumulative 
Proportion 
of  Total 
Benefit 

New  Training  R&D 

23,417 

1.000 

Folksonomies 

14,042 

0.988 

HPI  in  the  Army 

11,792 

0.978 

Training  Support  Tools 

11,292 

0.971 

Improvisation 

10,292 

0.947 

Critical  Thinking  Skills 

9,854 

0.934 

Leader  Socialization 

8,604 

0.891 

Predictive  Models 

8,063 

0.870 

Life-long  Learning 

7,313 

0.835 

Team  Task  Analysis 

6,313 

0.783 

The  GEL  Model 

5,313 

0.726 

R&D  on  dL 

5,063 

0.708 

Knowledge  Management 

4,563 

0.667 

Experience-Based  Learning 

3,563 

0.585 

Blended  Learning 

3,063 

0.537 

Training  Capabilities 

2,875 

0.518 

Diagnostic  Evaluation 

2,500 

0.474 

Lessons  Learned 

1,750 

0.380 

Implementation  Plan  for  dL 

1,250 

0.314 

Analyses  of  Training 

542 

0.180 

Tailored  Training 

250 

0.114 

B-l 


Table  B-2 

Cumulative  FYDP  Costs  and  Benefits 


Short  Titles 

Cumulative 
FYDP 
Costs 
(K  Dollars) 

Cumulative 
Proportion 
of  Total 
Benefit 

New  Training  R&D 

76,417 

1.000 

Folksonomies 

29,542 

0.988 

Improvisation 

21,292 

0.978 

HPI  in  the  Army 

20,104 

0.965 

Training  Support  Tools 

19,604 

0.958 

R&D  on  dL 

18,104 

0.934 

The  GEL  Model 

15,604 

0.893 

Life-long  Learning 

14,854 

0.875 

Critical  Thinking  Skills 

12,854 

0.823 

Leader  Socialization 

11,271 

0.780 

Knowledge  Management 

10,729 

0.760 

Lessons  Learned 

8,729 

0.677 

Predictive  Models 

7,229 

0.611 

Diagnostic  Evaluation 

6,479 

0.575 

Team  Task  Analysis 

4,729 

0.481 

Analyses  of  Training 

3,729 

0.425 

Training  Capabilities 

2,771 

0.358 

Experience-Based  Learning 

2,146 

0.314 

Blended  Learning 

1,563 

0.266 

Implementation  Plan  for  dL 

1,375 

0.248 

Tailored  Training 

500 

0.114 

B-2 


